39 Why Numerical Methods for Likelihood

39.1 Challenges

Frequentist model:

\[X_1, X_2, \ldots, X_n {\; \stackrel{\text{iid}}{\sim}\;}F_{{\boldsymbol{\theta}}}\]

Bayesian model:

\[X_1, X_2, \ldots, X_n | {\boldsymbol{\theta}}{\; \stackrel{\text{iid}}{\sim}\;}F_{{\boldsymbol{\theta}}} \mbox{ and } {\boldsymbol{\theta}}\sim F_{\boldsymbol{\tau}}\]

Sometimes it’s not possible to find formulas for \(\hat{{\boldsymbol{\theta}}}_{\text{MLE}}\), \(\hat{{\boldsymbol{\theta}}}_{\text{MAP}}\), \({\operatorname{E}}[{\boldsymbol{\theta}}| {\boldsymbol{x}}]\), or \(f({\boldsymbol{\theta}}| {\boldsymbol{x}})\). We have to use numerical methods instead.

39.2 Approaches

We will discuss the following numerical approaches to likelihood based inference:

• Expectation-maximization (EM) algorithm
• Variational inference
• Markov chain Monte Carlo (MCMC)
  • Metropolis sampling
  • Metropolis-Hastings sampling
  • Gibbs sampling