Clustering

Study guides: Exam MAS-II, Exam DISC-DA

Clustering is an unsupervised Statistical Learning technique that groups observations into clusters so that observations within a cluster are similar to each other and dissimilar from those in other clusters. There is no response variable Y — the goal is to discover structure in \mathbf{X}.

Two main families:

Hierarchical clustering algorithm:

1. Start with each observation in its own cluster (n clusters)
2. Merge the two most similar clusters using a linkage criterion
3. Repeat until all observations are in one cluster
4. Cut the dendrogram at a chosen height to obtain K clusters

Linkage methods (how inter-cluster distance is defined):

• Complete: maximum distance between any pair of points in the two clusters

• Single: minimum distance (can produce chaining)

• Average: average distance between all pairs across clusters

• Centroid: distance between the two clusters' centroids (mean vectors); can produce inversions, where two clusters fuse at a height below either of them in the dendrogram

• Clustering requires a dissimilarity measure (e.g., Euclidean distance, correlation-based distance); whether to standardize the variables (to standard deviation one) depends on the application: it gives every variable equal weight, and is advisable when they are measured on different scales

• Used in actuarial science for policyholder segmentation and territory grouping

• Choices that change the answer (James et al., §12.4.3): whether to standardize, which dissimilarity measure and linkage to use, where to cut the dendrogram, and how many clusters to ask K-means for. Each can change the clusters, so try several and keep the most useful and interpretable result. See Similarity and Distance Measures

• Clusters are always found: Clustering returns clusters whether or not the data contain real subgroups. Whether they would reappear in an independent sample is a hard question, so clusters used as segments should be checked on other data and against what the business knows

Choosing the Number of Clusters An actuary applies hierarchical clustering to 500 policyholders using vehicle and driver characteristics. The dendrogram shows large height gaps between 3 and 4 clusters. What does this suggest?

Answer A large gap in the dendrogram between merging 4 clusters into 3 indicates that those 4 clusters are naturally distinct — merging them requires combining very dissimilar groups. This suggests K = 4 clusters is a natural choice.

How it appears on the exam

• DISC-DA: Section 8.4 covers segmenting data with cluster analysis. Questions describe policyholder or claim data with no target and ask which technique groups them, what standardizing the features would change, or how to read a dendrogram or choose the number of clusters.
• MAS-II: Questions apply K-means and hierarchical clustering mechanics, such as linkage choices and the effect of scaling, from James et al. Chapter 12.

Exam traps

• Clustering is not classification: There is no response variable. A method trained on labeled outcomes is supervised, not cluster analysis.
• Unscaled features: On raw data, the feature with the largest numbers dominates the distances and so the clusters.
• Single linkage chains: It tends to string observations into long, unbalanced clusters; average and complete linkage give more balanced ones.