© 2016 IBM Corporation

Trends in Model Risk GovernanceIan Francis

IBM Watson Financial ServicesRisk & Compliance Innovation Forum

London

24 May 2017

6/7/172

Where are we headed and who’s setting the destination?

Where are we right now?

How are we going to get there?

Where are we headed

Regulators Risk Management Accountants

Direction being set by main parties

6/7/17

SR-11-7

FRTB

TRIM

CPS-220

IFRS 9 OSFI-E23

Supervisory Guidance on Model Risk Management (SR 11-7/OCC 2011-12)

CCAR

"BHCs (bank holding companies)should ensure that they have sound model risk management, including independent review and validation of all models used in internal capital planning, consistent with existing supervisory guidance on model risk management (SR letter 11-7)."(Source: Comprehensive Capital Analysis and Review 2015: Summary Instructions and Guidance)

The board should direct senior management to provide information about the firm'sestimation approaches, model overlays, and assessments of model performanceA firm should independently validate or otherwise conduct effective challenge of modelsused in internal capital planning, consistent with supervisory guidance on model riskmanagement. The model review and validation process should include an evaluation ofconceptual soundness of models and ongoing monitoring of the model performance(source: SR15-18)

• The ECB’s objective is to harmonise the use of the internal model approaches of banks placed under its jurisdiction

• Targeted review of internal models spread over several years. • Complemented by quantitative impact studies (QISs)• The ultimate aim of the work is to establish a level playing field for the use

of internal model approaches.

TRIM – Targeted Review of Internal Models

FRTB Governance Requirements

Model Governance Desk Representation, Nomination / Assessment Risk Factor Assessment

Internal Risk Transfer Governance Overall FRTB Regulatory Compliance

AustraliaAPRA – GPS 112,CPS 220

CanadaOSFI – E23

Additional Global References

The rapid growth in regulatory compliance activity over Models has in turn increased the spend by firms on:

ü Documentation and Development activitiesü Resources to cover enhanced Model validation

requestsü Resources to interpreting requirements across

multiple jurisdictions ü Resources to apply processes against multiple

regimes ( both change and BAUrequests)

Increase in Costs

Summary of where we are going

Desired Regulatory Outcomes

•• Increased Model transparency / understanding

•• Improved documentation•• 1st line Model Risk ownership•• Evidence of effective

challenge•• Robust and independent 2nd

and 3rd line activities

Desired Institutional Outcomes

•• Efficiency of process , outcome

•• Improved communications between Model Risk Stakeholders ( 1st and 2nd

Line)•• Regulatory compliance•• Improved Insight and

Reporting

Where are we now?

Where are we now

Culture Methodology Technology

Regionally

Model Register with Meta dataDocumentation

SimpleCapturing Model Inputs / OutputsAssessing Model RiskViewing and managing Model interdependenciesDynamic Workflow

AlignedLinkage with Data governanceIntegration with Model Development toolsIntegration for Model Deployment and Operational control

Integrated

Maturity Stages – Inventory Management

Recording basic Model Validation outcomes

SimpleManually recording Model metrics• Model Performance• Model Operations• Governance Framework

scoresResolution activities as a result of Model ValidationValidation resources

AlignedIntegration with Models for automated performance metricsUse of MRG tools to schedule and resource plan Model Validation ActivitiesData lineage and quality early warning

Integrated

Maturity Stages– Monitoring & Review

19

Standard Model Risk Management Organization

Chief Risk Officer

Head of Model Risk

Management

Head of Market Model

Validation

Head of Credit Model

Validation

Head of Misc. Validation

Head of Model Governance

Title Roles and Responsibilities

Chief Risk Officer Responsible for overseeing program and approving procedures

Head of Model Risk Management

• Managing program• Managing policies and procedures, including proposing revisions• Communication with BOD, including reporting• Responsible for representing the group internally• “Stature” requirement• May have complete or partial model approval authority• Interfacing with regulators

Head of Model Governance

• Policy and procedure maintenance and drafting• Reporting design and execution• Model risk tiering• Scheduling/PMO• Issue and remediation tracking

Head of Market Model Validation

• Focused on validation of market risk –type models, valuation models based on market variables.

• Choice of validation groupings should be made for greatest clarity and efficiency.

Head of Credit Model Validation

• Focused on validation of credit risk models where credit is defined as default, rather than market risk

Head of Operational Risk, Capital, etc. Model Validation

• Focused on validation of other models as determined, which may include operational risk models, CRR models, etc.

Progress – A view from the Fed/OCC

Governance

• Significant attention and discussion to evaluate current MRM practices & develop remediation plans

• Healthy Scepticism about Model Outputs & Limitations• Attention to monitoring the entire cycle of development, implementation,

use, and validation

Model Design

• Firm-wide standards for model design, as well as more detailed procedures to guide development of specific model types

• Clear documentation about the original model design and processing components – allows third party to follow the path of development

• Evidence that model developers have conducted research about industry/academic theory & practice relating to the model

Validation • Increased adoption of “parallel validation” with enhanced safeguards

Vendor Models• Clearly understand model limitations and assumptions of vendor models

to determine whether the model is appropriate for the intended use• Updates to vendor methodology, data, etc are tracked and reviewed

Source: OCC / FRB Presentation: Richmond Federal Reserve Bank MRM Forum - https://www.richmondfed.org/-/media/richmondfedorg/conferences_and_events/banking/2016/mrmforum2016_day1slides.pdf

How are we getting to our destination

6/7/1721

Understanding of the road maps

Turn by Turn navigation

Continual updates on performance

Traffic updates and re-routing

Overview of Progress

So how do we get the most efficient / effective route

Suitable Inventory

Documented , clear process

Ongoing Monitoring

Real-time / Robust Issue Management

6/7/17

A good inventory is based on a solid foundation

Elements of a good inventory

ü Completeness and transparency – using a consistent principles-based model definition

ü Careful definition of useü Robust metadata collectionü Connection to issues tracking;

validation activities and documentation

Model Risk Management Activities

Development Review Model Use

Monitoring Remediation Change Management

Dimension to Address for an Effective MRM Process

Culture Systems

Process Organisation

Numerous benefits to those can establish an ongoing monitoring programüPotential Cost benefitsüOperational effectiveness – reduction in manual effort / errorsüEarly insight

Ongoing Monitoring

Inventory

Model Validation Tools

Model Documentation

Model Performance

Results

Model inputs defined and models used for multiple usages

Automated tests are set up in the initial validation

Automated derivative pricing validation

Select relevant existing test from library, including model wrapper and test parameters – can be kicked off manually or automated

Model wrapper code

Graphical and numerical results output for analysis

Implementation (Unit) Testing:

Calibration Round Trip

Tests how well calibration performs when the data is setup for a perfect solution.

PDE Test

Tests how well the model evaluates its expectation value.

Smoothness Test

Tests for dangerous oscillations, or excess noise, which can ruin greeks, calculations.

Benchmark Test

Compares pricing and greeks to those of one or more alternative implementations of the same model.

Backtesting:

Calibration Error TestTests how well the model’s faces fit real market surfaces.

Calibration StabilityTests how stable the model’s calibration is.

Hedge Performance TestTests how well the model’s hedge removes risk in real market scenarios.

PnL Attribution TestTests how well the model’s hedge removes risk in real market scenarios.

Challenge Model TestCompares pricing and greeks to those found in a dissimilar, market standard model.

Laddered Stress TestObserves behavior under application of a stress with increasing (laddered) intensity.

Automated Tests run Regularly

Email alerting / advanced warnings

Output reports for analysis and stored within governance system

Dashboards for high-level visibility

Actionable Insight

What else is on the road

Opportunity for Machine Learning, AI

Watson Financial Services / Presentation Title / Date37

Centralisation

Thank you