Model Risk Management and CECL - Store & Retrieve Data … · 2016-09-19 · Model Risk Management...

©2016 Crowe Horwath LLP

Model Risk Management and CECL

September 14, 2016

Mike Budinger, Principal

©2016 Crowe Horwath LLP 22

Discuss model validation and key items when

building a plan 02Model Validation

Preparing for the CECL transition and considerations of

model validation03

Model Validation

& CECL

Review key principles of the model risk

management guidance 01MRM

Contents for Discussion


MRM


Regulatory Guidance on Model Risk Management

The Office of the Comptroller of the Currency (OCC) and the Federal Reserve published Supervisory Guidance on Model

Risk Management in April 2011 (OCC 2011-12, SR 11-7).

Three Primary Areas of Focus:

Model Development

Implementation, and Use

Model Validation

Model Governance,

Policies, and Controls


Understanding Models

What are model risks? The potential for adverse consequences from decisions based on models that are incorrect or

misused

Safety and soundness of financial system

Alignment with accounting standards

Compliance with laws and regulations

Business and strategic decisions

InputsComplex

TransformationFrequent Use Model


Evolution of MRM in Banking

Source: Model Risk Management and the Great Financial Crisis: The Rise of Modern Model Risk Management


Model Inventory

provides a central

repository to store all the

information related to

various Models

Model requirements,

design, implementation

plan, business purpose

and use need to be clearly

documented and executed

according to sound

practices

Model Validation is essential

to verify the accuracy and

effectiveness against a

Standard Model or any

available benchmarks

A prescriptive calibration

methodology and model

performance monitoring

and analysis are

necessary to ensure the

ongoing reliability and to

optimize the overall

efficiency of the Models

Prevailing Internal Controls

are required to govern the

Model lifecycle with the

Bank’s Board of Directors

providing oversight

The rings of the Crowe Model Risk Management framework capture the core requirements

of the Supervisory Guidance and expand on key principles

Model Risk Management Framework

Source: Crowe Horwath LLP


Identifying & inventorying,

assessing risk, and

managing accordingly

Calibrating models over time

based on feedback and

richer data sets

Monitoring model

performance and

benchmarking

Implementing robust governance

processes including policies &

internal audit

Limiting model use to

purpose for which it was

developed

Executing sound validation

process by competent,

independent parties

Managing Model Risk


Gaining Enterprise Buy-In

Adopting Tools and

Templates

Developing Metrics

Facilitating Validations

Recruiting Resources

Maturing the Curve: Moving from Policy to Practice


Regulatory Pressure for MRM

Identifying and Leveraging Enabling Technology

Source: Crowe Horwath LLP Webinar, January 2016


Model Validation


Set of activities

intended to

verify the model

is performing as

expected

PerformancePerformed by

independent

staff with

appropriate

incentives,

competence,

and influence

IndependenceAll components,

including input,

processing, and

reporting, are

subject to

validation

ComponentsValidation

activities should

continue on an

ongoing basis

after a model

goes into use

Ongoing

Essential Elements of Model Validation


Model Risk Criteria Weight Model Risk

Materiality - An errors impact on the financial statement 20%

Complexity - The level of complication of the model. 20%

Maturity -The level of automation, integration, and repeatability. 10%

Environment - The level of controls in place, data limitations,

and resource availability.10%

Regulatory Impact - The exposure to regulatory review and

model implications.20%

Reporting - The use of the model in support of business activity.20%

Model Risk Score 4.0

Model

Risk

Score

1- 2 Low

2 - 4 Moderate

4+ High

Assessing Model Risk – ALLL Example

5

5

3

1

4

4


Validation Segments

Each model segment has unique model risks.

These model risks, along with the associated

control environment, require testing and

effective challenge to assess the effectiveness

of the model

Design & Development

Input Processing Output & Use

Implementation

Performance

Identifying Model Risks


Risk ID Risk Segment Description

MR01Design and

DevelopmentThe loan portfolio is not segmented in a manner

necessary to capture homogenous risk.

MR02 Input ProcessingPortfolio balances and charge-off inputs are not

reconciled to the general ledger.

MR03 Implementation

The model(s) built and implemented do not align with

the conceptual design, creating gaps between the

deployed model and the stated business and

technical requirements.

MR04 Output and Use

Estimated credit losses do not change in a

directionally consistent manner with other credit

quality indicators.

MR05 PerformanceOngoing monitoring not established to understand

when model should be redeveloped.

Model Validation Plan – By Risk Segment

©2016 Crowe Horwath LLP

16

ReportingConsolidation

of observations

and gaps

ApproachAdopting a

structured

approach can

go a long way

towards

success

Prior Validations

Don’t be afraid

to peek at prior

validations

Staffing Matters

Not all models

are the same

IndependenceApproach with

caution!

Additional Model Validation Considerations


Model Validation and CECL

Transition


Making the CECL Transition

Governance and Oversight

Enabling

Technology

Data Inventory

Resource

Capabilities.

Risk Identification

Accounting and

Regulatory Alignment

Assesses the ability of the

CECL model to meet

accounting and regulatory

needs and objectives.

Accounting

and

Regulatory

Alignment


Risk Identification and Model Validation

•Understanding the main drivers of portfolio performance

•Loan Structures

•Lending Attributes

•Prepayment Risk

•Relationship with Macroeconomic Factors

•Establishes segmentation and modeling considerations

Model Validation Considerations

•Does the model(s) capture homogenous risk?

• Is the conceptual design in alignment with accepted industry practice?

•Exploration of alternatives?


Data Inventory and Model Validation

•Understanding the availability of data

•Reliability

•Missing Data

•Proxy Data

•Time Horizon

•Retention of older data can be a limitation – archiving


•What are the known assumptions or exclusions about data that might affect the model and/or reporting?

•How is data transformed from the source prior to reaching the model?


Resource Capabilities and Model Validation

•Understanding the capabilities and limitations of human resources to develop and execute on the CECL model

•Staffing Availability

•Portfolio and Organizational Knowledge

•Accounting Understanding

•Technical Competence


•Are the individuals reviewing the model output different than those developing?

• Is there a backup plan to operate the model in the event of key personnel departure?


Enabling Technology and Model Validation

•Understanding the existing systems, including the capabilities and limitations of those systems

•Source Systems/Data Warehouse

•Modeling Applications

•Financial Statement Spreading

•Third Party Vendor Applications

•Financial Statement Report Generation


•What are the controls in place to limit access to the model and changing settings?

•How is the conceptual design integrated into the modeling system and business setting?

•What is the support provided by third parties for model development and execution?


Governance and Oversight and Model Validation

•Understanding risk management practices surrounding the development, execution, and maintenance of the

CECL model

•Established roles and responsibilities of the board and

senior management

•Policies and procedures

•Model risk management principles and practices

are in play


•How does effective challenge of the model and results take place?

• Is reporting on the model clear and comprehensive, including model performance?

•What are the plans to provide ongoing monitoring over the model?


Determine the ability of management to promptly prepare

the ALLL financial reporting estimate to accurately reflect the

operations and results of the institution

Readiness

Determine that management has established effective internal controls to meet financial reporting requirements

Internal Controls

Determine that policies and procedures have been drafted and implemented to effectively estimate the ALLL

Policies and Procedures

Successful transition to the CECL-based ALLL methodology will

require the active involvement of the audit committee, the board

of directors, and senior management

Governance and Oversight – Additional Thoughts


Accounting

Credit Risk

Internal Audit

Lending

IT

Model Risk

Governance and Oversight – Transition Readiness


Year 1 Year 2 Year 3

Risk Identification

Data Inventory

Resources

Enabling Tech

Gov & Oversight

Evaluate What Drives Credit Losses Scenario Modeling

Risk Factor Reassessment

Historical Data Collection

Resource Capability Continuous Assessment

Develop Plan to Fill Shortfalls

Data Aggregation and Management

Develop Roadmap

Educate Testing of ICoFR

Data Analysis

Create Teams

Identify Shortfalls

ICoFR Design

Determine Risk Profile

Create Supplemental Data

Risk Factor Reassessment

Data Collection (Years 2 and 3)

Modeling Approach

Model Development and Calibration

Develop and Finalize Policies

Model Validation

Develop FS Disclosures

Data Validation

Roadmap to Readiness – An Example


Final Thoughts - Model Risk in the CECL Transition

•Model risk management can have a seat at the table

•Model risk can support third party vendor selection process

•Incremental validation during development can provide immediate feedback

•Schedules should be established for validation

•Alignment of documentation to model risk management standards is a key to successful validation


In accordance with applicable professional standards, some firm services may not be available to attest clients.

This material is for informational purposes only and should not be construed as financial or legal advice. Please seek guidance specific to your organization from qualified advisers in your jurisdiction.

© 2016 Crowe Horwath LLP, an independent member of Crowe Horwath International crowehorwath.com/disclosure

Mike Budinger

Phone 216.623.7517

[email protected]

Thank you

http://www.crowehorwath.com/about/legal/

Model Risk Management and CECL - Store & Retrieve Data … · 2016-09-19 · Model Risk Management...

Documents

Transcript of Model Risk Management and CECL - Store & Retrieve Data … · 2016-09-19 · Model Risk Management...