Actuarial Notes

'Machine Decisions': Governance of AI and Big Data Analytics

By CRO Forum · 2019 · CP312-111-25

Study guide: Exam CP-312

A CRO Forum paper on how insurers can adapt existing governance and model risk frameworks to cover Big Data tools, including machine learning algorithms. It examines what reasonable steps can be taken to know what is happening inside the black box, ensure reliability and avoid unfair discrimination. It focuses solely on the (re)insurance industry and is meant as a practical overview for risk managers.

On the syllabus

1 Executive summary

  • 1.1 Introduction
    • Big Data and AI present particular challenges due to complexity, self-calibration, autonomy and the potential for unexpected results and unforeseen impacts.
  • 1.2 A number of key insights emerged
    • It is vital to invest sufficient effort upfront to define the problem the tool is expected to solve in pursuit of a business objective.
    • Human intervention is and will remain critical; a key challenge is identifying when and how humans need to intervene and what triggers prompt their intervention.
    • Training needs apply at all levels in an organization.
    • Model governance techniques and frameworks that exist today do not need to be fundamentally altered, but can be enhanced and adjusted to meet the needs of complex tools and machine learning.

2 Background and approach taken

  • 2.1 Approach and observations about use
  • 2.2 Reputational risk
  • 2.3 Regulatory interest

3 Summary of types of Big Data Analytics in use

  • 3.1 Differences traditional regression and machine learning
  • 3.2 Weighting the pros and cons of the Big Data Analytics algorithms
  • 3.3 Tools for insight and validation
    • 3.3.1 Global variable importance
    • 3.3.2 Partial dependency plots
    • 3.3.3 Prediction explanations: LIME
    • 3.3.4 SHapley Additive exPlanations (SHAP)
    • 3.3.5 Other techniques

4 Key learnings from case studies

  • 4.1 Problem statement
    • 4.1.1 Explanation of key principle
    • 4.1.2 Rationale why this key principle is important
  • 4.2 Humans in the loop
    • 4.2.1 Explanation of principle
    • 4.2.2 Rationale why this principle is important
  • 4.3 Training needs
    • 4.3.1 Explanation of key principle
    • 4.3.2 Rationale why this key principle focussing on training needs is important

5 Applying an ethical framework to the use of Big Data and AI in insurance

  • 5.1 Managing ethical risks
  • 5.2 Ethical purpose
  • 5.3 Means
  • 5.4 Outcomes

6 Model Management Framework for Big Data Analytics

  • 6.1 Modelling process and associated risks
  • 6.2 Model governance

7 Summary and key conclusions

Appendix 1 Sample checklist of model governance questions for Big Data Analytics tools

  • 1.1 Design, specification and implementation
  • 1.2 Calibration and operation
  • 1.3 Decision making

Sources