Variable Selection

Study guides: Exam ATPA, Exam MAS-I, Exam MAS-II, Exam PCPA

Variable Selection is the process of choosing which predictors to include in a model's Model Structure, balancing improved fit against the risk of overfitting and loss of interpretability.

Method	Description
Best subset selection	Fit every combination of the $p$ predictors; keep the best model of each size (lowest RSS, or lowest Deviance for a GLM), then choose among the sizes as in the last row
Forward stepwise	Start with no predictors; at each step add the one that most improves the fit; choose among the resulting sizes as in the last row
Backward stepwise	Start with all predictors (needs $n > p$ ); at each step remove the least useful one; choose among the sizes as in the last row
Hybrid stepwise	Add predictors as in forward stepwise, but after each addition remove any that no longer improve the fit
Lasso (Regularization)	Fit all $p$ predictors with a penalty that sets some coefficients exactly to zero, so the selection happens within the fit; the penalty $\lambda$ is chosen by Cross-Validation
Choosing among sizes	Compare models with different numbers of predictors via AIC (or $C_p$ ), BIC, adjusted $R^2$ or Cross-Validation error — not by raw RSS, $R^2$ or deviance, which always improve as predictors are added

How many models are fitted. Best subset selection fits all $2^p$ models; forward and backward stepwise each fit only $1 + p(p+1)/2$ . With $p = 20$ that is $1{,}048{,}576$ models against $211$ . The price is that stepwise selection is not guaranteed to find the best subset: if the best one-predictor model is $\{x_1\}$ but the best two-predictor model is $\{x_2, x_3\}$ , forward stepwise cannot reach it, because every later model must keep $x_1$
Raw Deviance still has two uses: comparing models with the same number of parameters (best subset's choice within each size), and comparing nested models through a deviance (F or likelihood-ratio) test. Compared raw across sizes it gives an unfair advantage to the model with more parameters
Lower AIC or BIC indicates a better trade-off between fit and complexity; BIC penalizes additional parameters more heavily than AIC, so it tends to favor sparser models
In a Linear Mixed Model, variable selection also requires deciding which effects are fixed versus random, not just which predictors to include
Variable selection should combine statistical significance (from the parameter estimate table) with business and actuarial judgment, e.g., regulatory restrictions on certain rating variables
Selecting too few variables causes underfitting (high bias); selecting too many causes overfitting (high variance) — a holdout sample or cross-validation helps detect overfitting that in-sample fit statistics alone would miss

Choosing Between Two Model Specifications Two competing GLMs for claim frequency: Model 1 has $5$ parameters and a log-likelihood of $-2{,}040$ . Model 2 adds $2$ more parameters and has a log-likelihood of $-2{,}037$ . Using AIC ( $\text{AIC} = -2\ell + 2k$ ), which model is preferred?

Answer $\text{AIC}_1 = -2(-2{,}040) + 2(5) = 4{,}080 + 10 = 4{,}090$ $\text{AIC}_2 = -2(-2{,}037) + 2(7) = 4{,}074 + 14 = 4{,}088$ Model 2 has the lower AIC ( $4{,}088 < 4{,}090$ ), so the extra $2$ parameters are worth the added complexity — Model 2 is preferred.