Modern Methods of Applied Statistics (Spring 2024) STAT 34800

Instructor: Aaron Schein
TAs: Jimmy Lederman, Sean O'Hagan, Jinwen Yang

Term: Spring 2024
The University of Chicago

---

Logistics:

• Time: Tuesday and Thursday, 3:30am-4:50pm
• Place: Eckhart room 133
• TA office hours:
  • Jimmy: Mon 1:30-2:30pm (Jones 304)
  • Sean: Wed 3:00-4:00pm (Jones 304)
  • Jinwen: Fri 10:00-11:00am (Jones 304)
• Instructor office hours:
  • Aaron: Thurs 5:00-6:00pm (Searle 236)

Assignments

• Assignment 1: Supervised learning. Due Saturday March 30 at 11:59pm on GradeScope.
• Assignment 2: Priors, regularization, shrinkage. Due Saturday April 6 at 11:59pm on GradeScope.
• Assignment 3: Exponential families, conjugacy, entropy. Due Monday April 15 at 11:59pm on GradeScope.
• Assignment 4: HMMs and the USS Scorpion. Due Wednesday April 24 at 11:59pm on GradeScope.
• Assignment 5: EM and Gibbs sampling. Due Friday May 10 at 11:59pm on GradeScope.

Schedule

Lecture 1 (March 19): Review of decision theory & supervised learning

• Reading / resources:
  • Materials for L1-L3 of Matthew Stephens' course: STAT 348 (Spring 2021)
  • Chap 2 of Elements of Statistical Learning
  • Chap 2 of An Introduction to Statistical Learning with Applications in Python
  • Chap 3 and 12 of Advanced Data Analysis from an Elementary Point of View
• Lecture materials:
  • Notebook
  • iPad notes

Lecture 2 (March 21): Review of decision theory & supervised learning

• Reading / resources:
  • Materials for L1-L3 of Matthew Stephens' course: STAT 348 (Spring 2021)
  • Chap 3 of Elements of Statistical Learning
  • Chap 5.1, 6.2, 7.1 of An Introduction to Statistical Learning with Applications in Python
  • Chap 7 of Advanced Data Analysis from an Elementary Point of View
• Lecture materials:
  • Notebook
  • iPad notes

Lecture 3 (March 26): Intro to Bayesian modeling & decision theory

• Reading / resources:
  • Scott Linderman's slides on Bayesian analysis of Gaussian models
  • Murphy (2007) "Conjugate Bayesian analysis of the Gaussian distribution"
  • Jeffrey Miller's slides on Bayesian linear regression
  • Chap 9 of "Mathematics for Machine Learning"
• Lecture materials:
  • Notebook
  • iPad notes

Lecture 4 (March 26): Intro to Bayesian modeling & decision theory

• Reading / resources:
  • Chap 1 of Berger (1985) Statistical Decision Theory and Bayesian Analysis
  • Chap 15 "The Navy Searches" of The Theory That Would Not Die
• Lecture materials:
  • Notebook
  • iPad notes

Lecture 5 (April 2): Conjugacy, exponential families, and information theory

• Reading / resources:
  • David Blei's lectures notes on conjugacy in exponential families
  • Jeffrey Miller's slides on conjugate priors
  • Chap 14.3 of John Duchi's lecture notes on exponential families as maximum entropy distributions
  • Chap 2.4-2.6 and 4.1-4.3 of Mackay (2005) Information Theory, Inference, and Learning Algorithms
• Lecture materials:
  • iPad notes

Lecture 6 (April 4): Information theory, compression, model selection

• Reading / resources:
  • Chap 4, 5.1-5.4, 8, 28 of Mackay (2005) Information Theory, Inference, and Learning Algorithms
  • Kirsch (2004) "On Bayesian Model Selection: The Marginal Likelihood, Cross-Validation, and Conditional Log Marginal Likelihood"
  • Fong & Holmes (2020) "On the marginal likelihood and cross-validation"
  • Gleick (2011) The Information
• Lecture materials:
  • iPad notes

Lecture 7 (April 9): Probabilistic graphical models (PGMs)

• Reading / resources:
  • Chap 2 of Michael Jordan's lecture notes
  • David Blei's lecture notes on basics of PGMs
• Lecture materials:
  • iPad notes

Lecture 8 (April 11): Inference in PGMs: variable elimination, belief propagation, and message-passing

• Reading / resources:
  • Chap 3 and chap 4 of Michael Jordan's lecture notes
  • David Blei's lecture notes on inference in PGMs
  • Yedidia et al. (2001) "Bethe free energy, Kikuchi approximations, and belief propagation algorithms"
• Lecture materials:
  • iPad notes

Lecture 9 (April 16): Learning and inference in hidden Markov models (HMMs)

• Reading / resources:
  • Chap 17 of Murphy (2012) "Machine learning: a probabilistic perspective" (available as e-book via the library)
  • Scott Linderman's slides on HMMs
• Lecture materials:
  • iPad notes

Lecture 10 (April 18): Learning and inference in hidden Markov models (HMMs)

• Lecture materials:
  • iPad notes

Lecture 11 (April 23): Bayesian mixture models and EM

• Reading / resources:
  • Chap 9 of Bishop (2006) Pattern Recognition and Machine Learning on mixtures and EM
  • Scott Linderman's slides on Bayesian mixtures
  • Scott Linderman's slides on EM
  • David Blei's lecture notes on Bayesian mixtures
• Lecture materials:
  • iPad notes

Lecture 12 (April 25): Gibbs sampling and MCMC

• Reading / resources:
  • Chap 11.1.6-11.3 of Bishop (2006) Pattern Recognition and Machine Learning
  • David Blei's lecture notes on Bayesian mixtures and Gibbs sampling
  • Matthew Stephen's lecture notes on Gibbs sampling
  • Scott Linderman's slides on MCMC
  • Geweke (2004): "Getting it Right: Joint Distribution Tests of Posterior Simulators"
  • Roger Grosse's blogpost on Geweke testing
• Lecture materials:
  • iPad notes

Lecture 13 (April 27): Poisson / non-negative matrix factorization and auxiliary-variable MCMC

• Reading / resources:
  • Dunson & Herring (2005) "Bayesian latent variable models for mixed discrete outcomes"
  • Gopalan et al. (2014) "Scalable Recommendation with Poisson factorization"
  • Lee and Seung (1999) "Learning the parts of objects by non-negative matrix factorization"
  • Gillis (2014) "The How and Why of Nonnegative Matrix Factorization"
• Lecture materials:
  • iPad notes

Midterm (May 2)

Lecture 14 (April May 7): Latent Dirichlet allocation (LDA) and variational inference (VI)

• Reading / resources:
  • Chap 27.3 of Murphy (2012) Machine Learning: A Probabilistic Perspective
  • Scott Linderman's slides on CAVI for LDA
  • Jeffrey Miller's slides on CAVI for LDA
  • Matt Gormley's slides on LDA
  • Blei, Ng, Jordan (2003) "Latent Dirichlet Allocation"
  • Pritchard, Stephens, Donnelly (2000) "Inference of Population Structure Using Multilocus Genotype Data"
• Lecture materials:
  • iPad notes

Lecture 15 (April May 9): Variational inference (cont.): CAVI and SVI

• Reading / resources:
  • Blei, Kucukelbir & McAuliffe (2017) "Variational inference: A review for statisticians"
  • Chap 10.1-10.2, 10.4 of Bishop (2006) Pattern Recognition and Machine Learning
  • Slides from lecture 2 of STAT 451 on CAVI and SVI
  • Scott Linderman's slides on VI
• Lecture materials:
  • iPad notes