Regularization

Study guides: Exam SRM, Exam PA, Exam MAS-I, Exam MAS-II

Regularization adds a penalty on coefficient size to the fitting criterion, shrinking estimates toward zero. It trades a little bias for a large reduction in variance — the Bias-Variance Tradeoff deliberately exploited.

\text{Ridge:}\quad \min_{\beta}\ \mathrm{RSS} + \lambda\sum_{j=1}^{p}\beta_j^{2}

\text{Lasso:}\quad \min_{\beta}\ \mathrm{RSS} + \lambda\sum_{j=1}^{p}\lvert\beta_j\rvert

\text{Budget form:}\quad \min_{\beta}\ \mathrm{RSS} \ \text{ subject to } \sum_{j=1}^{p}\beta_j^{2} \le s \ \text{(ridge)} \ \text{ or } \sum_{j=1}^{p}\lvert\beta_j\rvert \le s \ \text{(lasso)}

• \lambda = 0 returns least squares; \lambda \to \infty drives every coefficient to zero. \lambda is chosen by Cross-Validation
• Budget s runs the other way. For every \lambda there is an s giving the same estimates: a large s is a small \lambda (an s big enough to contain the least squares fit is \lambda = 0), a small s a large \lambda. With p = 2 the lasso region \lvert\beta_1\rvert + \lvert\beta_2\rvert \le s is a diamond and the ridge region \beta_1^2 + \beta_2^2 \le s a circle; the RSS contours often first touch the diamond at a corner on an axis, which is why the lasso returns exact zeros
• Ridge (\ell_2) shrinks all coefficients smoothly and keeps every predictor — good when many predictors each contribute a little. With correlated predictors it shrinks their coefficients toward each other rather than choosing one; the two simple remedies for collinearity itself are to drop one of the collinear variables or combine them into a single predictor
• Lasso (\ell_1) sets some coefficients exactly to zero, so it performs Variable Selection as part of the fit — good when the truth is sparse
• Elastic net mixes the two penalties, keeping lasso's selection while handling correlated predictors like ridge
• Predictors must be standardized first: the penalty is not scale-invariant, so a variable measured in thousands would otherwise be penalized differently from one measured in units
• The intercept is left unpenalized

Ridge or Lasso? A rating model has 200 candidate telematics variables and 3,000 policies. The actuary believes only a handful actually matter and wants a deployable model. Which penalty, and what does the tuning look like?

Answer Lasso. The belief is sparsity, and lasso's \ell_1 penalty zeroes out the irrelevant variables, producing a model with a few predictors that can be filed and explained. Ridge would keep all 200 with small coefficients. Standardize the predictors, fit over a grid of \lambda, and pick \lambda by 10-fold CV — the one-standard-error rule if a simpler model is worth a little accuracy.