Variational Analysis of Empirical Risk Minimization

Andrew Nobel

This talk presents a variational framework for the asymptotic analysis of empirical risk minimization in general settings.  In its most general form the framework concerns a two-stage inference procedure.  In the first stage of the procedure, an average loss criterion is used to fit the trajectory of an observed dynamical system with a trajectory of a reference dynamical system.  In the second stage of the procedure, a parameter estimate is obtained from the optimal trajectory of the reference system.  I will show that the empirical risk of the best fit trajectory converges almost surely to a constant that can be expressed in variational form as the minimal expected loss over dynamically invariant couplings (joinings) of the observed and reference systems.  Moreover, the family of joinings minimizing the expected loss fully characterizes the asymptotic behavior of the estimated parameters.  I will illustrate the variational framework through an application to the well-studied problem of maximum likelihood estimation, and the analysis of system identification from quantized trajectories subject to noise, a problem in which the models themselves exhibit dynamical behavior across time.  As time permits, I will give an overview of new results in a more Bayesian setting, specifically Gibbs posterior estimation of Gibbs distributions.