Maximum Likelihood Estimation

Study guides: Exam VEE-MS, Exam FAM, Exam ALTAM, Exam ASTAM, Exam MAS-I, Exam MAS-II

Maximum Likelihood Estimation (MLE) is a method of parameter estimation that chooses the parameter value $\hat{\theta}$ that maximizes the likelihood function — the joint probability (discrete data) or joint density (continuous data) of the observed sample, read as a function of the parameter.

$L(\theta) = \prod_{i=1}^n f(x_i \mid \theta)$

$\hat{\theta}_{\text{MLE}} = \arg\max_\theta L(\theta) = \arg\max_\theta \ell(\theta)$

$\ell(\theta) = \ln L(\theta) = \sum_{i=1}^n \ln f(x_i \mid \theta)$

Taking the log turns the product into a sum and simplifies maximization
When the maximum is at an interior point and the log-likelihood is differentiable, the MLE solves the score equation $\ell'(\hat{\theta}) = 0$ . The maximum can instead sit on a boundary: for $\text{Uniform}(0, \theta)$ , $L(\theta) = \theta^{-n}$ on $\theta \ge \max x_i$ is decreasing, so $\hat{\theta} = \max X_i$ , where $\ell'(\hat{\theta}) = -n/\hat{\theta} \ne 0$
Under regularity conditions — every $f(x \mid \theta)$ has the same support, the true $\theta$ is an interior point, the log-likelihood is differentiable, and the score equation has a unique root — MLEs are consistent, asymptotically normal with variance $1/[nI(\theta)]$ (see Fisher Information), and asymptotically efficient (achieve the Cramér-Rao Lower Bound in large samples). $\text{Uniform}(0, \theta)$ , whose support depends on $\theta$ , is the standard exception
Invariance property: if $\hat{\theta}$ is the MLE of $\theta$ , then $g(\hat{\theta})$ is the MLE of $g(\theta)$

Adjustments for incomplete data:

Censoring: observation is known to exceed some value $c$ ; contribute $P(X > c \mid \theta) = S(c \mid \theta)$ to the likelihood
Truncation: only observations exceeding a threshold $d$ are recorded; condition on $X > d$ , contributing $f(x \mid \theta)/S(d \mid \theta)$

MLE for an Exponential Distribution Five claim sizes (in $ $1{,}000$ s) are: $2, 3, 5, 4, 6$ . Assuming $X_i \sim \text{Exp}(\theta)$ , find the MLE of $\theta$ .

Answer For $\text{Exp}(\theta)$ : $f(x) = \frac{1}{\theta}e^{-x/\theta}$ , so $\ell(\theta) = -n\ln\theta - \frac{1}{\theta}\sum x_i$ . Setting $\ell'(\theta) = -\frac{n}{\theta} + \frac{\sum x_i}{\theta^2} = 0$ gives $\hat{\theta} = \bar{x} = \frac{2+3+5+4+6}{5} = 4$ (i.e., $ $4{,}000$ ).