39 Why Numerical Methods for Likelihood

39.1 Challenges

Frequentist model:

\[X_1, X_2, \ldots, X_n {\; \stackrel{\text{iid}}{\sim}\;}F_{{\boldsymbol{\theta}}}\]

Bayesian model:

\[X_1, X_2, \ldots, X_n | {\boldsymbol{\theta}}{\; \stackrel{\text{iid}}{\sim}\;}F_{{\boldsymbol{\theta}}} \mbox{ and } {\boldsymbol{\theta}}\sim F_{\boldsymbol{\tau}}\]

Sometimes it’s not possible to find formulas for \(\hat{{\boldsymbol{\theta}}}_{\text{MLE}}\), \(\hat{{\boldsymbol{\theta}}}_{\text{MAP}}\), \({\operatorname{E}}[{\boldsymbol{\theta}}| {\boldsymbol{x}}]\), or \(f({\boldsymbol{\theta}}| {\boldsymbol{x}})\). We have to use numerical methods instead.

39.2 Approaches

We will discuss the following numerical approaches to likelihood based inference:

  • Expectation-maximization (EM) algorithm
  • Variational inference
  • Markov chain Monte Carlo (MCMC)
    • Metropolis sampling
    • Metropolis-Hastings sampling
    • Gibbs sampling