CAP 6610, Machine Learning, Fall 2019
Place:CSE Building; E222
Time:MWF 4 (10:40-11:30 a.m.)
Instructor:
Arunava Banerjee
Office: CSE E336.
E-mail: arunava@ufl.edu.
Phone: 294-6641.
Office hours: Tuesday 2:00 p.m.-4:00 p.m.
TA:
Johnathan Smith
Office: CSE E555.
E-mail: emallson@ufl.edu.
Office hours: Wednesday 2:00 p.m.-4:00 p.m.(at CSE E309)
Pre-requisites:
- The official pre-requisites for this course is COT5615 (Mathematics for
Intelligent Systems). Specifically, knowledge of calculus and linear algebra
is necessary since we shall be touching on mathematical probability theory.
In addition, proficiency in some programming language is a must.
Textbook (recommended): Machine Learning: A Probabilistic Perspective,
Murphy, ISBN-10: 0262018020.
Reference: Pattern Recognition and Machine Learning,
Bishop, ISBN 0-38-731073-8.
Reference: Pattern Classification, 2nd Edition, Duda, Hart
and Stork, John Wiley, ISBN 0-471-05669-3.
Tentative list of Topics to be covered
- Review of mathematical probability theory; finite sample probability
bounds.
- Decision Trees
- Bayes decision theory
- Bayesian learning
- Maximum likelihood estimation and Expectation Maximization
- Linear and generalized linear models for regression and classification,
- Sparsity promoting priors with conjugates and their relationship to regularization
- Kernel methods including Support Vector Machines
- Error Back-propagation and Neural Networks
- Mixture models
- Hidden Markov models
- Principal Components Analysis
- Independent Components Analysis
- Reinforcement Learning
- Performance evaluation: re-substitution, cross-validation, bagging, and boosting
The above list is tentative at this juncture and the set of topics we end up
covering might change due to class interest and/or time constraints.
Please return to this page at least once a week to check
updates in the table below
Evaluation:
- One individual project spanning the semester: 15%
- Homework assignments (written and programming): 10%
- Three midterm exams: 25% each (1 hr, in-class)
- There will be no makeup exams (Exceptions shall be made for those that
present appropriate letters from the Dean of Students Office).
The final grade will be on the curve.
Course Policies:
- Late assignments: All homework assignments are due before class.
- Plagiarism: You are expected to submit your own solutions to the
assignments. While the final project and presentation will be done in groups,
each member will be required to demonstrate his/her contribution to the work.
- Attendance: Their is no official attendance requirement. If you
find better use of the time spent sitting thru lectures, please feel free to
devote such to any occupation of your liking. However, keep in mind that it is
your responsibility to stay abreast of the material presented in class.
- Cell Phones: Absolutely no phone calls during class. Please turn
off the ringer on your cell phone before coming to class.
Academic Dishonesty:
See http://www.dso.ufl.edu/judicial/honestybrochure.htm
for Academic Honesty Guidelines. All academic dishonesty cases will be
handled through the University of Florida Honor Court procedures as
documented by the office of Student Services, P202 Peabody Hall. You may
contact them at 392-1261 for a "Student Judicial Process: Guide for Students"
pamphlet.
Students with Disabilities: Students requesting classroom
accommodation must first register with the Dean of Students Office. The Dean of
Students Office will provide documentation to the student who must then provide
this documentation to the Instructor when requesting accommodation.
Announcements
Please return to this page at least once a week. All
announcements will be posted on this page.
The three midterms will take place on: Sept 23rd, Oct 28th, and Dec 4th.
Midterms are NOT cumulative.
HomeWorks
| HomeWork |
Due Date |
Solutions |
List of Topics covered
| Week |
Topic |
Additional Reading |
| Aug 18 - Aug 24 |
- Introduction
- Examples of ML applications and what they do
- Spam fliter, Ad-sense, Face detection/recognition, Hurricane path
prediction, Stock prediction, web search, Recommendation system
|
|
| Aug 25 - Aug 31 |
- Putative framework:
- Supervised, Unsupervised Learning. Reinforcement Learning
- Labeled/unlabeled datasets, training/testing.
- Over-fitting to training data
- Hypothesis space/ Concept class
- Multi-variate regression and normal equation; Ordinary least squares
|
|
Sep 01 - Sep 07 |
- Labor day and Hurricane Dorian
- Ridge regression/Tikhonov regularization/weight decay
- Lasso(least absolute shrinkage and selection operator)
|
|
| Sep 08 - Sep 14 |
-
The "Risk functional" approach
Loss function, Hypothesis space/Concept class
- Application to the classification problem, and the 0/1 loss function
- Decision theory
|
|
| Sep 15 - Sep 21 |
-
The "Risk functional" approach continued
- Application to regression and density estimation
- Empirical Risk and the Empirical risk minimization principle
- Jensen's inequality
- Brief review of Mathematical Probability Theory;
- Sample space, outcome
- Measurable space, sigma algebra
- Random Variables; Distribution function
- Expected Value
|
|
| Sep 22 - Sep 28 |
- MIDTERM I
- Probability bounds: Markov and Chebyshev inequalities
- Weak law of large numbers
- Hoeffding's inequality
|
Here Proof of the VC theorem
|
| Sep 29 - Oct 05 |
- Hoeffding's inequality continued
- Vapnik Chervonenkis theorem for generalization error
- VC dimension
- Shatter coefficient (sometimes called growth function)
|
| Oct 06 - Oct 12 |
- Artificial sigmoidal neuron and gradient descent on error
- Multi-layer perceptrons and Error back propagation
- Neural nets as universal approximators
- The vanishing and exploding gradient problem
- Loss functions; Activation Functions
|
- Book On Deep Learning
by Goodfellow, Bengio, Courville
- GAN by Goodfellow et al.
- VAE by Kingma and Welling.
|
| Oct 13 - Oct 19 |
- Recap of Error backpropagation. On-line learning, epoch, over-fitting,
convolutional neural net. Transpose convolutional net.
- Autoencoders
- Brief review of GAN (Generative Adversarial network) and VAE
(Variational Auto encoder) objectives
- Convex functions, Thm: local minima = global minima
- Convex sets.
|
|
| Oct 20 - Oct 26 |
- The Lagrange Multiplier technique; Equality and Inequality constrints
- Convex optimization: Inequality and Equality constraints
- Primal form of maximal margin classifier (aka SVM)
- With and Without slack formulation
|
|
| Oct 27 - Nov 02 |
- MIDTERM II
- Duality; the Lagrange Dual problem.
- Dual formulation of maximal margin classifier (aka SVM)
- Linear, Polynomial, Gaussian Kernel
|
|
| Nov 03 - Nov 09 |
- Strong duality
- Slater's condition
- Intuitive explanation of dual formulation
- Decision Trees; Gini Impurity, Information theoretic Entropy
- Prunning, cross validation, minimum description length
|
|
| Nov 10 - Nov 16 |
- Mutual Information
- Kullback-Liebler Divergence
- Unsupervised Learning
- Frequentist (Maximum Likelihood)
- Bayesian (Prior and Posterior distribution; maximum a posteriori)
- Notion of Posterior consistency
|
|
| Nov 17 - Nov 23 |
- Maximum likelihood (ML) and Bayesian parameter estimation
(maximum a posteriori, i.e., MAP)
- Conjugate priors, Binomial and it conjugate (Beta)
- Multinomial and Dirichlet
- Multivariate Gaussian
- Started Mixture of Gaussian and Expectation Maximization
|
|
| Nov 24 - Nov 30 |
- Mixture of Gaussians and Expectation Maximization.
- Thanksgiving break
|
- Wiki on
K-Means clustering.
- Here are D'Souza's notes.
|
| Dec 01 - Dec 07 |
- Finished Mixture of Gaussians and Expectation Maximization.
- K-Means clustering
- MIDTERM III
|