Actuarial Notes

Adding Value with Model Validation

By Winston Tuner Hall, Michael Minnes and Veltcho Natchev · Society of Actuaries · 2017

Study guide: Exam CP-312

An article from The Modeling Platform, Issue 6 (November 2017), in which three MassMutual practitioners define what they believe a model validation should entail and the value proposition it offers. It also covers how to gain stakeholder buy-in and how to reach consensus with stakeholders on issues, findings and mitigations.

On the syllabus

Definition of Model Validation

  • A validation is a comprehensive review of the entire model environment, not just a deep dive into the accuracy of calculations, programming code logic or data inputs; its scope includes the process used to develop and implement the model and the artifacts created during its life cycle.
  • The main objectives are testing a model's conceptual soundness and continued fit for purpose, including identification of potential risks and limitations, as an effective challenge to the existing production model.
  • Validation effort should be commensurate with the model's risk.
  • Consistent with the three-lines-of-defense concept, the most effective validation combines the first, second and third lines of defense: model owners (first), an independent validator from risk management (second), and Internal Audit's post-validation follow-up (third).

Model Validation's Value Proposition

  • Effect Culture Change
    • A validation should start a shift in the actuarial workforce from routine production tasks toward continuous improvement and analysis, by challenging model owners to document the model's "fit for purpose", develop ongoing monitoring such as scenario or stress testing, and develop validation controls.
  • Mitigate Key-Person Risk
    • Key-person risk includes the risk that a model's control framework depends heavily on key persons; validation mitigates it by verifying fit for purpose, controls that evidence accuracy of results, documentation that lets another modeler use or replicate the model, and ongoing performance monitoring.
  • Prepare for PBR
    • Principle-based reserving requires model owners to describe material decisions, disclose experience study and assumption governance, and describe the approaches used to validate models.
  • Balance Costs
    • Costs are lower when development teams involve Internal Audit and Model Validation early; for models already in production the costs can be significant but are mostly short term.

Obtaining Stakeholder Support and Buy-In

  • Ensure There is Appropriate "Tone at the Top" Regarding Validations
  • Generate Shared Hopes With All Model Stakeholders About the Process
    • Use a consistent set of standards for validation planning, execution and results reporting, and a single scale for rating models across the organization.
  • Build Validator's Credibility With Model Stakeholders
  • Introduce/Reinforce and Continue Driving Value of "Lines of Defense" Concept
  • Whenever Possible, Avoid Disruption of Business
  • Create a Collaborative Environment
    • A validation's objective is not to look for a "gotcha" opportunity but to provide comfort to stakeholders and identify improvement opportunities.
    • An escalation hierarchy and a mechanism for "risk acceptance" by model owners and business area management are agreed up front.

Achieving Consensus: Issues, Mitigations and Findings

  • Findings and recommended mitigations are discussed with the model owner before they are presented to management, and recommendations are limited to those that are actionable and implementable.
  • The validator provides full transparency through frequent communication, and a fully documented, shareable record of all validation steps, including a final validation report with feasible, practical recommendations.
  • Internal Audit may take an active part in managing post-validation deliverables, while model governance tracks major themes emerging across validations.

What Should You Do?

  • Undertake a holistic model validation approach through a multipronged effort among the first, second and third lines of defense, and use model validation to help move a company's culture from a routine task-oriented one to a value-add and analytical one.

Sources