27 MLE Examples: One Sample
27.1 Exponential Family Distributions
Suppose \(X_1, X_2, \ldots, X_n\) are iid from some EFD. Then,
\[ \frac{\partial}{\partial \eta_k} \ell(\boldsymbol{\eta} ; \boldsymbol{x}) = \sum_{i=1}^n T_k(x_i) - n \frac{\partial}{\partial \eta_k} A(\boldsymbol{\eta}) \] Setting the second equation to 0, it follows that the MLE of \(\eta_k\) is the solution to
\[ \frac{1}{n} \sum_{i=1}^n T_k(x_i) = \frac{\partial}{\partial \eta_k} A(\boldsymbol{\eta}). \]
where note that \(\frac{\partial}{\partial \eta_k} A(\boldsymbol{\eta}) = {\operatorname{E}}[T_k(X)]\).
27.2 Summary of MLE Statistics
In all of these scenarios, \(Z\) converges in distribution to Normal\((0,1)\) for large \(n\).
| Distribution | MLE | Std Err | \(Z\) Statistic |
|---|---|---|---|
| Binomial\((n,p)\) | \(\hat{p} = X/n\) | \(\sqrt{\frac{\hat{p}(1-\hat{p})}{n}}\) | \(\frac{\hat{p} - p}{\sqrt{\frac{\hat{p}(1-\hat{p})}{n}}}\) |
| Normal\((\mu, \sigma^2)\) | \(\hat{\mu} = \overline{X}\) | \(\frac{\hat{\sigma}}{\sqrt{n}}\) | \(\frac{\hat{\mu} - \mu}{\hat{\sigma}/\sqrt{n}}\) |
| Poisson\((\lambda)\) | \(\hat{\lambda} = \overline{X}\) | \(\sqrt{\frac{\hat{\lambda}}{n}}\) | \(\frac{\hat{\lambda} - \lambda}{\sqrt{\hat{\lambda}/n}}\) |
27.3 Notes
- For the Normal and Poisson distributions, our model is \(X_1, X_2, \ldots, X_n\) iid from each respective distribution
- For the Binomial distribution, our model is \(X \sim \mbox{Binomial}(n, p)\)
- In the Normal model, \(\hat{\sigma} = \sqrt{\frac{\sum_{i=1}^n (X_i - \overline{X})^2}{n}}\) is the MLE of \(\sigma\)
- The above formulas were given in terms of the random variable probability models; on observed data the same formulas are used except we observed data lower case letters, e.g., replace \(\overline{X}\) with \(\overline{x}\)
27.4 Binomial
Approximate \((1-\alpha)\)-level two-sided CI:
\[\left(\hat{p} - |z_{\alpha/2}| \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}, \hat{p} + |z_{\alpha/2}| \sqrt{\frac{\hat{p}(1-\hat{p})}{n}} \right)\]
Hypothesis test, \(H_0: p=p_0\) vs \(H_1: p \not= p_0\):
\[z = \frac{\hat{p} - p_0}{\sqrt{\frac{\hat{p}(1-\hat{p})}{n}}} \mbox{ and } \mbox{p-value} = {\rm Pr}(|Z^*| \geq |z|)\]
where \(Z^*\) is a Normal\((0,1)\) random variable.
27.5 Normal
Approximate \((1-\alpha)\)-level two-sided CI:
\[\left(\hat{\mu} - |z_{\alpha/2}| \frac{\hat{\sigma}}{\sqrt{n}}, \hat{\mu} + |z_{\alpha/2}| \frac{\hat{\sigma}}{\sqrt{n}} \right)\]
Hypothesis test, \(H_0: \mu=\mu_0\) vs \(H_1: \mu \not= \mu_0\):
\[z = \frac{\hat{\mu} - \mu_0}{\hat{\sigma}/\sqrt{n}} \mbox{ and } \mbox{p-value} = {\rm Pr}(|Z^*| \geq |z|)\]
where \(Z^*\) is a Normal\((0,1)\) random variable.
27.6 Poisson
Approximate \((1-\alpha)\)-level two-sided CI:
\[\left(\hat{\lambda} - |z_{\alpha/2}| \sqrt{\frac{\hat{\lambda}}{n}}, \hat{\lambda} + |z_{\alpha/2}| \sqrt{\frac{\hat{\lambda}}{n}} \right)\]
Hypothesis test, \(H_0: \lambda=\lambda_0\) vs \(H_1: \lambda \not= \lambda_0\):
\[z = \frac{\hat{\lambda} - \lambda_0}{\sqrt{\frac{\hat{\lambda}}{n}}} \mbox{ and } \mbox{p-value} = {\rm Pr}(|Z^*| \geq |z|)\]
where \(Z^*\) is a Normal\((0,1)\) random variable.
27.7 One-Sided CIs and HTs
The one-sided versions of these approximate confidence intervals and hypothesis tests work analogously.
The procedures shown for the \(\mbox{Normal}(\mu, \sigma^2)\) case with known \(\sigma^2\) from last week are utilzied with the appropriate subsitutions as in the above examples.