Texts:

1. Required: Information Theory, Inference and Learning Algorithms, David J. C. MacKay, Cambridge University Press; 1st edition, 2002.
   
2. Recommended: Statistical Learning Theory, Vladimir N. Vapnik, Publisher: John Wiley and Sons, New York, 1998.
   
3. Other Material: Notes and papers from the following: Neural Computation

Office hours: MWF 3-4PM or by appointment. 

1. Homeworks: 25%.
2. Two Midterms: 25% each.
3. Project: 25%

1. Prerequisites: Machine Learning (CAP 6610)  or Neural Networks for Computing (CAP 6615). A familiarity with basic concepts in calculus, linear algebra, and probability theory. A partial list of basic requirements follows. Calculus: Differentiation, chain rule, integration. Linear algebra: Matrix multiplication, inverse, pseudo-inverse. Probability theory: Conditional probability, Bayes rule, conditional expectations. 
2. Homeworks/programs will be assigned bi-weekly. If you do not have any prior numerical computing experience, I suggest you use MATLAB for the programs.
3. A set of informal notes which will evolve with the course can be found here.
   

   
   

Probability, Entropy and Inference: entropy, basics of information theory, statistical mechanics and information geometry, information divergence measures, Fisher information, Riemannian metrics, manifold learning and nonlinear dimensionality reduction

Expected and Empirical risk: law of large numbers and statistical inequalities, learning and generalization, kernel methods

Bayesian networks: belief propagation, Bethe and Kikuchi approximate inference, Markov Chain Monte Carlo (MCMC)