Efficiency

Study guides: Exam MAS-I, Exam CP-312

Efficiency compares unbiased estimators by their variance relative to the smallest variance achievable: an unbiased estimator is efficient if it attains the Cramér–Rao lower bound, and of two unbiased estimators of the same parameter, the one with smaller variance is relatively more efficient (for an unbiased estimator the variance is the Mean Square Error).

$\text{Relative Efficiency}(\hat{\theta}_1, \hat{\theta}_2) = \frac{\text{Var}(\hat{\theta}_2)}{\text{Var}(\hat{\theta}_1)}$

$\text{Efficiency}(\hat{\theta}) = \frac{1/\big(nI(\theta)\big)}{\text{Var}(\hat{\theta})} \leq 1$

$I(\theta)$ is the Fisher Information in a single observation, so $1/\big(nI(\theta)\big)$ is the Cramér–Rao lower bound (CRLB) for a sample of size $n$ ; an unbiased estimator with $\text{Efficiency}(\hat{\theta}) = 1$ attains the CRLB and is called efficient
The bound, and with it $\text{Efficiency} \leq 1$ , holds only under regularity conditions — in practice, the support of $f(x \mid \theta)$ must not depend on $\theta$ . For $\text{Uniform}(0,\theta)$ it fails: $\frac{n+1}{n}X_{(n)}$ is unbiased with variance $\frac{\theta^2}{n(n+2)}$ , below the $\theta^2/n$ the formula would give, an "efficiency" of $n+2 > 1$
If $\text{Var}(\hat{\theta}_2) > \text{Var}(\hat{\theta}_1)$ , then $\hat{\theta}_1$ is more efficient — relative efficiency above $1$ favors $\hat{\theta}_1$
Relative efficiency compares two unbiased estimators (asymptotic relative efficiency, two asymptotically unbiased ones). If either is biased, compare their MSEs instead: $\text{MSE} = \text{Var} + \text{Bias}^2$ , so the smaller variance need not mean the smaller MSE
Under regularity conditions (a support that does not depend on $\theta$ , a true $\theta$ inside the parameter space, a differentiable log-likelihood), MLEs are asymptotically efficient: their variance converges to the CRLB as $n \to \infty$ , even when no finite-sample estimator attains it exactly
Efficiency is distinct from Consistency and Unbiasedness: an estimator can be consistent and unbiased yet inefficient if a lower-variance unbiased alternative exists

Comparing Two Unbiased Estimators For $X_1, \ldots, X_n \stackrel{\text{iid}}{\sim} N(\mu, \sigma^2)$ , consider the sample mean $\bar{X}$ and the sample median $M$ as estimators of $\mu$ . Both are unbiased, with $\text{Var}(\bar{X}) = \sigma^2/n$ and $\text{Var}(M) \approx 1.571\,\sigma^2/n$ for large $n$ . Which is more efficient?

Answer $\text{Relative Efficiency}(\bar{X}, M) = \frac{\text{Var}(M)}{\text{Var}(\bar{X})} = \frac{1.571\,\sigma^2/n}{\sigma^2/n} = 1.571$ Since this exceeds $1$ , $\bar{X}$ is the more efficient estimator — it has smaller variance for the same sample size.

Checking MLE Efficiency Against the CRLB For $X_i \stackrel{\text{iid}}{\sim} \text{Exp}(\theta)$ , the CRLB for any unbiased estimator of $\theta$ is $\theta^2/n$ , and $\hat{\theta}_{\text{MLE}} = \bar{X}$ has $\text{Var}(\bar{X}) = \theta^2/n$ . Is the MLE efficient here?

Answer $\text{Efficiency}(\bar{X}) = \frac{\theta^2/n}{\theta^2/n} = 1$ The MLE's variance exactly equals the CRLB for every $n$ , so $\bar{X}$ is efficient (not merely asymptotically efficient) in the exponential case.