Instructors:

Jeff Bilmes, Electrical Engineering and Marina Meila, Department of Statistics.

Course description:
Information theory deals with encoding data in order to transmit it correctly and effectively. Statistics and machine learning deal with estimating models of data and predicting future observations. Is there any relationship between the two? It turns out, perhaps not surprisingly, that the most compact encoding of the data is by the probabilistic model that describes it best. In other words, there is a fundamental link between information and probability.

We will start with the basic notions of information theory and explore its relationship to statistics and machine learning through a series of readings and discussions of classic articles. Among the topics that we intend to cover: Huffman coding, entropy, mutual information and Kullback-Leibler divergence; Maximum Likelihood, Bayesian inference and the Kullback-Leibler divergence; the maximum entropy principle; the information bottleneck method; the Fisher information; the method of types and basics of large deviation theory.

We will meet for two hours every week. The format will be that of a reading group: each time two participants prepare and lead the discussion of a paper or a book chapter. Credit will be given for participation. The first meeting will take place on Tuesday March 27 at 5pm in Padelford C-301. At that meeting we will decide the time and place for the rest of the course; we envisage it taking place in the late afternoon (i.e after 4:30). If you cannot attend the first meeting but would like to participate, send mail (before March 27) to one of the instructors with the times and days that would suit you.