38 Empirical Bayes
38.1 Rationale
Under the scenario that \boldsymbol{X} | \theta {\; \stackrel{\text{iid}}{\sim}\;}F_{\theta} with prior distribution \theta \sim F_{\tau}, we have to determine values for \tau.
The empirical Bayes approach uses the observed data to estimate the prior parameter(s), \tau.
This is especially useful for high-dimensional data when many parameters are simultaneously drawn from a prior with multiple observations drawn per parameter realization.
38.2 Approach
The usual approach is to first integrate out the parameter to obtain
f(\boldsymbol{x} ; \tau) = \int f(\boldsymbol{x} | \theta) f(\theta ; \tau) d\theta.
An estimation method (such as MLE) is then applied to estimate \tau. Then inference proceeds as usual under the assumption that \theta \sim f(\theta ; \hat{\tau}).
38.3 Example: Normal
Suppose that X_i | \mu_i \sim \mbox{Normal}(\mu_i, 1) for i=1, 2, \ldots, n where these rv’s are independent. Also suppose that \mu_i {\; \stackrel{\text{iid}}{\sim}\;}\mbox{Normal}(a, b^2).
f(x_i ; a, b) = \int f(x_i | \mu_i) f(\mu_i; a, b) d\mu_i \sim \mbox{Normal}(a, 1+b^2).
\implies \hat{a} = \overline{x}, \ 1+\hat{b}^2 = \frac{\sum_{k=1}^n (x_k - \overline{x})^2}{n}
\begin{align*} \operatorname{E}[\mu_i | x_i] & = \frac{1}{1+b^2}a + \frac{b^2}{1+b^2}x_i \implies \\ & \\ \hat{\operatorname{E}}[\mu_i | x_i] & = \frac{1}{1+\hat{b}^2}\hat{a} + \frac{\hat{b}^2}{1+\hat{b}^2}x_i \\ & = \frac{n}{\sum_{k=1}^n (x_k - \overline{x})^2} \overline{x} + \left(1-\frac{n}{\sum_{k=1}^n (x_k - \overline{x})^2}\right) x_i \end{align*}