Model Risk Management: Quantitative and qualitative aspects

Financial Institutions www.managementsolutions.com

Model Risk ManagementQuantitative and qualitative aspects

Design and LayoutMarketing and Communication DepartmentManagement Solutions - Spain

PhotographsPhotographic archive of Management SolutionsFotolia

© Management Solutions 2014All rights reserved. Cannot be reproduced, distributed, publicly disclosed, converted, totally or partially, freely or with a charge, in any way or procedure, without the expresswritten authorisation of Management Solutions. The information contained in this publication is merely to be used as a guideline. Management Solutions shall not be heldresponsible for the use which could be made of this information by third parties. Nobody is entitled to use this material except by express authorisation of ManagementSolutions.

Content

Introduction

Executive summary

Elements of an objective MRM framework

Model risk quantification

4

8

18

Model risk definition and regulations 12

26

Bibliography 36

Glossary 37

MANAGEM

ENT SO

LUTIONS

Model Risk Man

agem

ent - Quantitative an

d qualitative aspects

4

Introduction

5

In recent years there has been a trend in financial institutionstowards greater use of models in decision making, driven inpart by regulation but manifest in all areas of management.

In this regard, a high proportion of bank decisions areautomated through decision models (whether statisticalalgorithms or sets of rules)1.

First, the last few years have seen an increase in the use ofautomated electronic platforms that execute trade commandswhich have been pre-programmed by time, price or volume,and can start without manual intervention, a system known asalgorithmic trading. As an example, an automated tradecommand that took place on May 6, 2010 resulted in a 4,100million-dollar “flash crash” of the New York Stock Exchange,which fell more than 1,000 points and recovered to the samevalue in only 15 minutes2.

Second, partly encouraged by Basel3 regulations, banks areincreasingly using decision models (consisting of statisticalalgorithms and decision rules) in their origination, monitoringand credit recovery processes. Thus, whether or not a loan isviable is determined by estimating the probability of default(PD) of the client. Similarly, banks monitor customer accountsand anticipate credit deterioration using automatic alertmodels, pre-classify customers and determine their creditlimits; and, in credit collections, they develop statistical profilesof delinquent customers in order to apply different recoverystrategies.

In the commercial area, customers are able to select a product’scharacteristics (loan amount, term and purpose, insurancecoverage, etc.) and the system makes a real-time decision onviability and price. In many cases, the model asks the customera number of questions and proactively makes the offer thatbest suits the customer (doing this manually would be a slowand complex process).

The use of valuation models for products and financialinstruments has become widespread in financial institutions, inboth the markets and the ALM business. Some classic examplesare Black- Scholes, CAPM4 and Monte Carlo valuation models.

Another area where the use of models is more and morefrequent is fraud and money laundering detection. Bank andregulators alike use models that identify fraudulent or moneylaundering-oriented transactions, which requires combiningstatistical customer profiling models (know your customer -KYC), transaction monitoring rules and black lists.

Also, customer onboarding, engagement and marketingcampaign models have become more prevalent. These modelsare used to automatically establish customer loyalty andengagement actions both in the first stage of the relationshipwith the institution and at any time in the customer life cycle.Actions include the cross-selling of products and services thatare customized to suit the client’s needs, within the frameworkof CRM5.

1MIT (2005).2SEC (2010).3BCBS (2004-06).4Capital asset pricing model.5Customer relationship management

MANAGEM

ENT SO

LUTIONS

Model Risk Man

agem



6

Other examples include the calculation of capital charges for allexposures (credit, market, operational, ALM, etc.) through theirindividual components; the quantification of a bank’s currentliquidity position, projected under different scenarios; theprojection of the balance sheet and income statement and theuse of stress testing models6; or the modeling of many keycomponents in business planning and development, such asoptimal bundle, customer and non-customer income or churn(Fig. 1).

The use of models brings undoubted benefits, including:

4 Automated decision-making, which in turns improvesefficiency by reducing analysis and manual decision-relatedcosts.

4 Objective decision-making, ensuring that estimated resultsare the same in equal circumstances and that internal andexternal information is reused, thus leveraging historicalexperience.

4 Ability to synthetize complex issues such as a bank’saggregate risk.

However, using models also involves costs and risk, some ofwhich are the following:

4 Direct resource costs (economic and human) anddevelopment and implementation time.

4 The risk of trusting the results of an incorrect or misusedmodel. There are specific and recent examples of this whichhave resulted in large losses7.

Model risk may thus be defined as «the potential for adverseconsequences based on incorrect or misused model outputand reports»8.

Model error may include simplifications, approximations,wrong assumptions or an incorrect design process; whilemodel misuse includes applying models outside the use forwhich they were designed9.

Model risk thus defined is potentially very significant and hascaptured the attention of regulators and institutions, whoseapproach ranges from mitigation via model validation to theestablishment of a comprehensive framework for active modelrisk management.

In the more advanced cases, this active management has beenformulated into a model risk management (MRM) frameworkthat sets out the guidelines for the entire model design,development, implementation, validation, inventory and useprocess.

This is substantiated by the fact that regulators, particularly inthe U.S., have started to require such frameworks – as stated inthe guidelines issued by the Federal Reserve System (Fed)10 andthe Office of the Comptroller of the Currency (OCC ) – whichare serving as a starting point for the industry.

Regulations do not discuss model risk quantification aspects indetail, except in very specific cases relating to the valuation ofcertain products, in which they even require model risk to beestimated through valuation adjustments (model risk AVAs11 )that may result in a larger capital requirement or in the possibleuse of a capital buffer for model risk as a mitigating factor in abroader sense, without its calculation being specified.

Against this background, this study aims to provide acomprehensive view of model risk management: its definition,nature and sources, related regulations and practicalimplications. With this in mind, the document is structured inthree sections that address three goals:

4 Introducing model risk by providing a definition, analyzingits sources and summarizing the most importantregulations on the subject.

4 Describing a desirable framework from which to approachmodel risk management in a practical way and based onexamples seen in financial institutions.

4 Advancing model risk quantification (and its potentialpractical application) through a quantitative exercise thatwill illustrate the impact of this risk.

6See Management Solutions (2013).7For example, the London Whale case, which caused losses over $6.2 billion toJPMorgan in 2012. This falls within the OTC derivatives market, which had anexposure of almost $700 trillion in June 2013; see BIS (2013); or the incorrectvaluation of risk in some derivative instruments, which was one of the causes ofthe subprime crisis in the US in 2008. 8OCC-Fed (2011-12).9Ibid.10Ibid.11Model risk additional valuation adjustments (AVAs), detailed in EBA (2013).

7

Source: average of several financial institutions

Fig. 1. Model cloud: the size of each term is proportional to the number of models whose objective is this term

MANAGEM

ENT SO

LUTIONS

Model Risk Man

agem



8

Executive summary

9

This section is intended to summarize the main conclusionsreached on model risk management in financial institutions(which are elaborated on in the appropriate sections of thisdocument).

Model risk definition and regulations

1. The use of mathematical models by financial institutions inmany areas is rapidly gaining ground. This brings significantbenefits (objectivity, automation, efficiency, etc.) but alsoentails costs.

2. Among these costs is model risk, understood as the loss(economic, reputational, etc.) arising from decisions basedon flawed or misused models.

3. Thus understood, model risk may arise from three basicsources: data limitations (in terms of both availability andquality); estimation uncertainty or methodological flaws inthe model’s design (volatility of estimators, simplifications,approximations, wrong assumptions, incorrect design, etc.);and inappropriate use of the model (using the modeloutside its intended use, failure to update and recalibrate,etc.).

4. There has been little regulatory activity on model risk and,with one exception, regulations in this area refer almostexclusively to the need to make valuation adjustments inderivatives, the requirement to cover all risks in the internalcapital adequacy assessment process (ICAAP12) or the use ofthe Basel III leverage ratio as a mitigating factor of modelrisk when estimating risk-weighted assets for thecalculation of capital requirements via internal models13.

5. The exception relates to the Supervisory Guidance onModel Risk Management14 published by the OCC and theU.S. Fed in 2011-12, which, for the first time, accuratelydefined model risk and provided a set of guidelinesestablishing the need for entities to develop a Board-approved framework to identify and manage this risk(though not necessarily quantify it).

6. These guidelines cover all phases of a model’s life cycle:development and deployment, use, validation, governancepolicies, control and documentation by all involved; thedocumentation aspect is particularly emphasized due to itsimportance in effective model validation.

7. Among the main requirements is the need to addressmodel risk with the same rigor as any other risk, with the

particularity that it cannot be removed, only mitigatedthrough critical analysis or “effective challenge”.

8. Regulators expressly indicate that, while expert modeling,robust model validation and a duly justified though notexcessively conservative approach are necessary elementsin model risk mitigation, they are not sufficient and shouldnot serve as an excuse for the industry not to continue towork towards improving models.

9. With regard to model risk organization and governance,regulators do not prescribe a specific framework, but theydo address the need to establish a clear distinctionbetween model “ownership”15, “control”16 and“compliance”17 roles.

10. The Board of Directors is ultimately responsible forapproving a framework for model risk management (MRM)and should be regularly informed about any significantmodel risk to which the entity could be exposed.

11. Finally, regulators emphasize that the fundamentalprinciple in model risk management is “effective challenge”,understood as critical analysis by objective, qualifiedpeople with experience in the line of business in which themodel is used, who are able to identify model limitationsand assumptions, and suggest appropriate changes.

12. This comprehensive approach to model risk is novel in theindustry, and the expected trend for the coming years isthat the industry will gradually take it on board andimplement the prescribed practices, as the most advancedinstitutions are already doing.

12Internal Capital Adequacy Assessment Process.13BCBS (2010-11).14OCC-Fed (2011-12).15Ownership: refers to the model “owner” or user.16Control: function that validates the model and sets limits on its use.17Compliance: includes the execution of processes that ensure the other two rolesare performed according to the established policies.

MANAGEM

ENT SO

LUTIONS

Model Risk Man

agem



10


13. Banks that are more advanced in this area have a model riskmanagement (MRM) framework that is outlined in adocument approved by the Board of Directors and detailsaspects relating to organization and governance, modelmanagement, etc.

14. With regard to organization and governance, the MRMframework is characterized by its cross-cutting nature (itinvolves several areas, such as Business lines, Risk, InternalAudit, Technology, Finance, etc.), by explicitly defining thethree roles prescribed by the regulator (ownership, controland compliance) and assigning them to specific functionsin the organization and, above all, by establishing a ModelRisk Management function whose responsibility is to createand maintain the MRM framework.

15. While in some organizations the MRM function includesInternal Validation, in others these two functions areseparate, but in all cases the MRM function deals with allmatters related to model governance: maintaining anupdated and global inventory of models, producing anddisseminating MRM policies, evaluating all of theinstitution’s models on an annual basis, etc.

16. The MRM role, usually led by a Model Risk Officer (MRO),often has the final say on the approval of any of theinstitution’s models.

17. With regard to model management, the MRM frameworkincludes aspects such as: (a) model inventory covering all ofthe organization’s models in all areas (risk, commercial ,finance), usually supported on a suitable technological toolthat keeps track of all changes and versions; (b) a modelclassification or tiering system according to the risk posedto the bank, which will determine the level of thoroughnessthat will be required in the monitoring, validation and

documentation of models; (c) complete and detaileddocumentation on each model, which will allow the modelto be replicated by a third party and to be transferred to anew modeler without loss of knowledge; and (d) a modelfollow-up scheme for the early detection of bothdeviations from target performance and model misuse, inorder to act accordingly.

18. Model validation is central to model risk management andits fundamental principle should be the critical, effectiveand independent questioning (challenging) of all decisionsmade in connection with model development, monitoringand use. The frequency and intensity of validation for eachmodel should be commensurate with its risk, measuredthrough its associated tier, and the validation process andoutcome should be fully documented.

19. Though model risk quantification is not required by theregulations (with the exception of what has already beenmentioned in the introduction), some organizations arebeginning to incorporate it into their model riskmanagement program in order to objectively identify andmeasure any potential impacts should this risk materialize.


20. Beyond the regulations, and for management purposes,some organizations have begun to work on the model riskquantification cycle, which consists mainly of three phases:identifying and classifying model risk sources: (1) Datadeficiencies, (2) Estimation uncertainty or flaws in themodels, and (3) Model misuse; estimating model risk inrespect of each source (output sensitivity to inputvariations); and mitigating model risk through theimplementation of appropriate measures.

11

21. In order to explain this process, a study has beenconducted in which model and parameter risk associatedwith credit risk models is estimated.

22. To illustrate the estimation of data deficiencies, the firstexercise examines the impact of insufficient information inthe most predictive variables of a scoring model forconsumer mortgages. It is observed that the presence oferrors in the model’s most predictive variables can eithersignificantly increase the default rate or raise theopportunity cost (reduce the volume of acquired business)if default is fixed at the same level.

23. The second exercise is aimed at analyzing model risk arisingfrom estimation uncertainty, using the confidence intervalsof the estimators. To do this, the confidence intervals of ascoring model, the PD calculation and the LGD calculation(separately and combined) are used as a starting point toanalyze how the capital requirement for a mortgageportfolio may become underestimated through thecombined effect of uncertainty in the score estimators, PDcalibration and LGD estimation.

24. Finally, the third exercise analyzes model misuse through amodel (a scoring model, in this case) that was not updatedfor 12 months after it was first developed. This is done bycomparing the model’s predictive power at the time ofconstruction with that observed a year later, and assessingthe impact the decision not to update it had on default aswell as on opportunity cost (lower business acquisitionlevels). The observed outcome is that failure to update themodel under analysis for 12 months reduced predictivepower by several percentage points. The consequence ofthis reduction in predictive power was an increase in thedefault rate, or alternatively a reduction in the volume ofacquired business (i.e. an increase in the opportunity cost)with default remaining at the same level.

25. In summary, model risk can have a very significantquantitative impact that can result in management makingpoor decisions or even underestimating the institution’scapital requirement or provisions. It is therefore desirable tohave a MRM framework in place and, where appropriate,develop robust model risk estimation techniques aimed atimplementing suitable mitigation techniques.

MANAGEM

ENT SO

LUTIONS

Model Risk Man

agem



12

Model risk definition and regulations

13

What is a model?

When analyzing model risk, the first question that may arise iswhat a model is and what it is not.

According to the Fed and the OCC18, the term “model” refers to“a quantitative method, system, or approach that appliesstatistical, economic, financial, or mathematical theories,techniques, and assumptions to process input data intoquantitative estimates.” It has three components (Fig. 2):

4 The inputs, which introduce hypotheses and assumptionsin the model.

4 A method to transform the inputs into estimates. Thismethod will generally rely on statistics to producequantitative estimates.

4 A reporting component that converts the estimates intouseful business information.

Always according to the Fed and the OCC, “the definition ofmodel also covers quantitative approaches whose inputs arepartially or wholly qualitative or based on expert judgment,provided that the output is quantitative in nature.”

As can be seen, the concept of model is broader than what apartial interpretation – more related to the mathematicalalgorithm in a strict sense – would suggest, and includes expertmodels, among others.

However, this definition leaves some room for interpretation.For example, under a definition such as the above, thefollowing would no doubt have to be considered as models:

4 An algorithm for the calculation of value-at-risk (VaR) inmarket risk using either a Monte Carlo or a historicalsimulation.

4 A score to calculate the probability of default (PD) of theloans in a portfolio using logistic regression.

4 Mechanisms for the valuation of exposures, assets,instruments, portfolios, derivatives, etc.

Conversely, under this definition, it would be questionablewhether the following could be considered as models:

4 Any simple data aggregation: sums, averages, standarddeviations, financial ratios, etc.

4 A constant annual growth projection based on a singlehistorical year-on-year change, without any other analysisbeing conducted.

Fig. 2. Model components

18OCC- Fed (2011-12).

insufficient sample size. An example of this would be usingappraised value at the date a contract was formalizedrather than at the latest date when feeding a model,because the latest date has not been stored in the databases.

2. Estimation uncertainty or model error in the form ofsimplifications, approximations, flawed assumptions orincorrect model design. Any of these can occur at any pointin model development, from design to implementation,which can lead to erroneous results that are not consistentwith the purpose and intended use of the model. Theyinclude estimator uncertainty (reflected in the confidenceintervals, which are often calculated but tend not to beused), and also the use of unobservable parameters, thelack of market consensus on the model’s functional form,and computational difficulties, among others.

3. Model misuse, which includes using the model for purposesother than those for which it was designed (such asdeveloping a rating model based on a specific portfolio andapplying it to a different portfolio, for instance fromanother country), and not re-estimating or re-calibratingthe model in a long time.

MANAGEM

ENT SO

LUTIONS

Model Risk Man

agem



14

4 A decision-making mechanism based solely on the value ofa variable; for instance, a simple acceptance or rejectionrule based on the LTV value.

It is, however, for each institution to decide the scope of whatshould constitute a model and therefore would be affected bymodel risk and the policies defined in connection with it. Thisscope will at times be unclear and will require a degree ofsubjective judgment.

Once the scope has been defined, a decision will also have tobe made as to the types of model that are to be analyzed (risk,commercial, financial projections, etc.).

Model risk: nature and sources

Models are simplified representations of reality. Thissimplification is inevitable given the complexity of therelationship between variables, and in any case it is a source ofrisk that needs to be identified, analyzed and managed like anyother risk in an institution.

Model risk, according to the Fed and the OCC, is defined as “thepotential for adverse consequences from decisions based onincorrect or misused model outputs and reports”19.

If data deficiencies are explicitly added to this (as they result inincorrect model output), the sources of model risk may beclassified into three groups (Fig. 3):

1. Data deficiencies in terms of both availability and quality,including data errors, lack of critical variables, lack ofhistorical depth, errors in the feeding of variables or

Fig. 3. Model risk sources

19Ibid.

15

Regulatory context

To date, there are few specific regulations covering model risk,and these regulations tend to be non-specific in both theirdefinition and the expected treatment. In addition to theguidance document issued by the Fed and the OCC, which willbe analyzed in detail, there are some regulatory references tomodel risk that can be characterized into three types (Fig. 4):

4 Valuation adjustments: these are regulations governingthe need to prudently adjust the valuation of specificproducts (especially derivatives) to cover any potentialmodel risk.

Although Basel II already indicated20 the need to makethese adjustments, the main regulatory milestone in thisdirection occurred in 2013 with the publication of the EBA’sRTS on Prudent Valuation, which for the first time providedspecific methodological guidelines on how to performthese adjustments. This is the only case where the explicitquantification of model risk is covered in the regulations, aswill be detailed further in this document.

4 Buffer capital linked to ICAAP21: both Basel II in its secondpillar and some local regulators22 transposing thisregulation address the need to hold capital for all risksconsidered significant by the organization, as part of theirinternal capital adequacy assessment process (ICAAP).

This is also the case in other ICAAP-related processes, suchas the U.S. stress testing exercise known as CCAR23, wherethe regulator also suggests the possibility of holding acapital buffer for model risk (though it does not require it),which in fact some entities do in practice.

Fig. 4. Main regulatory references to model risk

However, most regulations either only mention model risktangentially or model risk can be understood to beincluded in “other risks”.

4 Other mentions: this encompasses other minor referencesto model risk, which treat it as being implicit in otherregulatory elements. The most notable case is the BaselCommittee’s treatment of model risk mitigation throughthe application of the leverage ratio, though it does not gofurther into detail24.

The main regulatory milestone, however, took place in 2011-12with the publication of the Supervisory Guidance on Model RiskManagement25 by the U.S. regulators. In this document, theconcept of model risk and the need for institutions to have aframework in place to identify and manage this risk is clearlydefined for the first time.

20BCBS (2004-06): “699. Supervisory authorities expect the following valuationadjustments/reserves to be formally considered at a minimum: […] whereappropriate, model risk.” 21Internal Capital Adequacy Assessment Process.22See, for example, Bank of Spain (2008-14).23Comprehensive Capital Analysis and Review. 24BCBS (2010-11): «16. […] the Committee is introducing a leverage ratiorequirement that is intended to achieve the following objectives: […] introduceadditional safeguards against model risk and measurement error bysupplementing the risk-based measure with a simple, transparent, independentmeasure of risk.” 25OCC-Fed (2011-12).

Fig. 5. Structure and scope of Fed and OCC’s rule on MRM

MANAGEM

ENT SO

LUTIONS

Model Risk Man

agem



16

To achieve this, the standard proposes a set of guidelines orprinciples for action on model risk, structured by area (Fig. 5),which can be summarized as follows (Fig. 6):

1. Model risk should be managed like any other risk; banksshould identify its sources and assess its magnitude inorder to manage it.

2. Model risk cannot be eliminated, only mitigated by goodmanagement. A combination of expert modeling androbust validation, while necessary, is not sufficient toeliminate model risk.

3. Consequently, there should be a framework for model riskmanagement (MRM) in place, approved by the Board ofDirectors.

4. A well supported conservative approach regarding inputs,outputs and model design is an effective tool, but shouldnot be an excuse not to continue to work towards thecontinuous improvement of models.

5. Prudent use of models may include duly justifiedconservative approaches, model stress testing, or a possiblecapital buffer for model risk. However, the regulator alsowarns that using too many conservative elements may leadto model misuse.

6. There are many sources of model risk and institutionsshould pay special attention to the aggregate model riskresulting from their combination.

7. While the choice of a suitable organizational model is at thediscretion of the institutions, a clear distinction needs to bemade between the roles of model ownership, control andcompliance; ownership involves knowing the model risk towhich the institution is subject; control deals with limitsetting and follow-up, as well as independent model

validation; and compliance is the set of processes thatensure the ownership and control roles are performed inaccordance with the established policies.

8. The Board of Directors is ultimately responsible for themodel risk management framework, which it must approve,and should be regularly informed about any significantmodel risk to which the entity is exposed.

9. The key principle in model risk mitigation is effectivechallenge: critical analysis by objective parties who arequalified and experienced in the line of business in whichthe model is used, and are able to identify the limitationsand assumptions and suggest appropriate improvements.

In summary, regulators are encouraging institutions to designmodel risk management frameworks that formalize the criteriato be followed in model development and implementation,ensure their prudent use, establish procedures to validate theirperformance and define policy governance and applicabledocumentation criteria.

This comprehensive approach to model risk is new in theindustry, and the expected trend for the coming years is thatinstitutions will gradually adopt it, as the more advancedorganizations are already doing.

17

Common sources of model riskModel risk is present in all stages of a model’s life cycle: development and implementation, monitoring, validation and audit; and stemsfrom three main sources: data, estimation uncertainty and error, and model use.

Some of the common problems that generate model risk are summarized below, by source type.

Development andimplementation

Use

Validation

Governance,policies and

control

Documentation

Fig. 6. Summary of the Fed and OCC guidelines on model risk management

4A clear definition of model purpose (statement of purpose) should be provided.4Data and other information should be appropriate, should undergo rigorous quality testing as well as checks to

ensure its use is justified, and any approximations should be thoroughly documented.4There should be testing procedures to assess the accuracy, robustness, stability and limitations of the model, as

well as a behavior test for different input values.4The entire development process should be documented in detail.

4User feedback provides an opportunity to check model performance.4Reports should be made using the estimator and model output values for different sets of inputs in order to

provide indicators of model precision, robustness and stability.4A prudent use of models may include duly justified conservative approaches, stress tests or a possible capital

buffer for model risk.4However, even the regulator warns that excessive use of conservative elements may lead to model misuse.

4The validation process should be embedded in a framework that:- Applies to all models, whether developed in-house or by third parties.- Applies to all model components (including inputs, estimation and outputs).- Is commensurate with model use, complexity and materiality.- Requires independence from model developers and users.

4Validators should have knowledge and expertise and should be familiar with the line of business in which themodel is used, and Internal Audit should verify the effectiveness of this framework.

4All models should be reviewed at least once a year and whenever they undergo material changes.

4The MRM framework, which must be reviewed annually, is developed by Senior Management and approved bythe Board, which should be regularly informed about any significant model risk.

4The three basic roles in MRM are: ownership (use of the model), control (model risk measurement, limit settingand monitoring) and compliance (compliance with policies by the other two roles).

4There must be a full inventory of all models in use.4It is advantageous to have external resources perform validation and compliance tasks.

4The model should be documented with a level of detail that would permit a third party to understand theoperation, limitations and key assumptions of the model.

4Where third party models (vendor models) are used, it should be ensured that adequate documentation isprovided so that the model can be properly validated.

1

2

3

4

5

Data- Error in data definition- Error in the mapping of data with the data

sources- Insufficient data feeding frequency- Problems arising from data provisioning - Data migrations- Accuracy of proxies (margin of error)- Insufficient sample- Insufficient historical depth - Lack of critical variables

Estimation- Estimator uncertainty - Model does not properly depict reality - Mathematical assumptions are inadequate- High sensitivity of expert adjustments - Use of unobservable parameters- Lack of market consensus- Computational difficulties- Non-use of confidence intervals- Outdated model due to parameter

decalibration, expert adjustments that are notupdated, variables with no discriminatingcapacity, etc.

- Model instability- Lack of comprehensive documentation- Insufficient analytical capabilities- Use of new methodology not supported by

academic research- Reduced Validation Unit independence from

model developers

Use- Using the model for purposes for which it was

not designed- Differences between regulatory and

management practices- Extending the model’s use beyond its original

scope (new products, markets, segments, etc.)- Model not effectively used in practice - Model not re-estimated or re-calibrated in a

long time- Unapproved changes introduced in the model- Differences in model definition and uses

between the commercial and the risk area- Model lacks credibility with the user

MANAGEM

ENT SO

LUTIONS

Model Risk Man

agem



18


19

MRM Policy

Several international financial institutions have developed anumber of key principles for model risk management (MRM),which are summarized and put forward in this section aspossible best practices.

All of these organizations believe it necessary to create a MRMframework that should be approved at the highest level and toestablish model ownership, control and compliance as the keyroles in model risk management.

This framework should be embodied in a written and explicitpolicy that has been approved by the Board of Directors andincludes four main elements (Fig. 7):

4 Organization and governance: description of roles andresponsibilities, in particular establishing the Model RiskManagement Function (MRM) as a reference for allmatters in this area.

4 Model management: guidelines on model classification,development, monitoring, documentation, inventory andreporting.

4 Model validation and change management: guidelinesfor the review of models, the approval of changes, andon waivers required for using the model prior toapproval.

4 Model risk quantification: methodology for model riskestimation, according to its nature and classification.

The Board of Directors is ultimately responsible for the approvalof the MRM framework. Also, and in line with the guidelinesestablished by the OCC and the Fed, the Board should receiveregular reports on the implementation of the MRM policy andmust be informed of any model risk that may have a materialimpact on the institution. Thus, best practice puts model risk onthe same level as any other significant risk for the institution.

From an organizational viewpoint, model risk management ischaracterized by three components:

4 Cross-cutting nature: model risk management affectsseveral areas within an institution, including the Businesslines, Risk, Finance, Internal Audit and Technology.

4 Roles: MRM policy should define the role of each area inmodel risk management. In line with OCC and Fedguidelines, three roles are usually defined:

- Ownership: this role lies with the model’s end-userareas, which are responsible for ensuring the model isproperly used as well as for reporting any errors orinconsistencies; the role of ownership also pertains to

Fig. 7 . Elements of a policy for model risk management (MRM)

MANAGEM

ENT SO

LUTIONS

Model Risk Man

agem



20

the model development area, which is close to the enduser. This role is normally carried out by the Businesslines, Finance or Risk.

- Control: an area that measures model risk, sets limitsand performs monitoring activities. This role is normallycarried out by the Risk area or by a specific andindependent control function which may or may notinclude the Internal Validation function.

- Compliance: an area that oversees compliance withpolicies by the other two roles. This role is normallyfulfilled by the Compliance or the Internal Auditfunction.

It should be noted that the model development process andthose involved in it are not necessarily the same in Europe as inthe U.S., where there is a tendency for schemes to be moredecentralized and embedded in the Business lines.

4 MRM Function: best practice includes the creation of aModel Risk Management function (MRM) directly reportingto the CRO26 and whose responsibility is to set up andmaintain the MRM framework. In some organizations MRMcontains the Internal Validation function (in others thesetwo functions are dealt with by separate units), andtherefore approves models and their use as well asmanaging all model governance-related issues. Itsresponsibilities include:

- Working closely with model owners in order to maintaina global, updated inventory.

- Validating models independently according to themodel’s classification (in some organizations, this role isperformed by other independent units).

- Approving the use of models and indicating theirlimitations.

- Conducting an annual assessment of all models in theinventory.

- Preparing and disseminating policy on model risk.

The MRM function tends to have a hierarchical structure withdifferentiated roles that report to the Model Risk Officer (MRO).

The MRM framework includes the guidelines that modelersneed to consider during the model development process, aswell as the key elements for model risk control. Some of themain guidelines are as follows:

1. Objective and use: all models should have a clearlyexplained objective and should be used for the purposethey were designed and approved for; any use outside theuses intended during development has to be expresslyapproved.

2. Inventory: Modelers must declare all models developed fortheir inclusion in the model inventory and for latervalidation and follow-up.

3. Tiering: all models should be classified according to the risktheir use involves for the organization.

4. Documentation: all models should be documented with alevel of detail proportionate to their tier, and shouldinclude a description of the objectives, intended uses, inputdata, hypothesis and methodology used.

26Chief Risk Officer.

21

5. No redundancy: prior to the development of a model, themodeler should confirm that there is not an existing modelthat may cover the user’s needs.

As regards model development and management componentsin an MRM framework, there are four key areas: modelinventory, tiering, documentation and follow-up. They aredetailed as follows.

Model Inventory

Banks should have a complete inventory of all existing models,with an aim to facilitate model risk governance andmanagement and to keep a record of all uses, changes and theapproval status of each model.

The model inventory should:

4 Include all entity models within the perimeter of what theMRM framework considers “model” and in all areas: Risk,Commercial, Finance, etc.

4 Include information on each model’s tier, itsdocumentation, its review status, intended and real uses,possible waivers that are being applied, and any otherinformation that the MRM function considers relevant forits good governance.

4 Be supported by a proper technological tool which includesa single and centralized repository for the whole entity and,ideally, an interface that permits dialogue between theowner (including the developer), the validator and theauditor of the model.

4 Keep track of all versions, changes, waivers, documents,considerations from the validators and the supervisor(when applicable), and expected dates to review andupdate the model.

As it can be appreciated, the construction and maintenance ofa model inventory requires a considerable effort byorganizations, but is an essential element of model riskmanagement.

Model Tiering

It is good practice to classify models according to the risk theiruse involves for the entity. Among other factors, thethoroughness required in model documentation, the need forvalidators to approve changes or the frequency and rigor of thefollow-up depend on this classification or tiering.

In a normal tiering process, the owner of a model suggests alevel of risk for the model, but the MRM function approves itand has the last word in this respect.

Model tiering is a partially subjective process that seeks toreflect the criteria through an institution estimates the risk ofeach model. As an example, tiering may depend on thefollowing factors (Fig. 8):

4 Materiality, which reflects the economic consequences of apossible error or wrong use of the model, and that eachowner quantifies in the most appropriate terms: exposureor balance-sheet volume of affected assets, margin at risk,reputational impact metrics, etc.

4 Sophistication, which expresses the model’s level ofcomplexity: highly complex mathematical formulation,dependency on a large number of input variables, observedstability of parameters, numerical approximations toanalytical expressions (e.g. stochastic differentialequations), new algorithms or algorithms for which there isno academic evidence of performance and stability, etc.

4 Impact on decisions, reflecting to what extent model resultshave an influence on decision processes that are sensitive

Fig. 8. Model tiering example

MANAGEM

ENT SO

LUTIONS

Model Risk Man

agem



22

for the institution, or on the financial statements andregulatory report. High dependency models are thosewhose result is the main axis of a key decision for theinstitution, such as the calculation of capital provisions,whereas low dependency models are those used as anotherfactor to support a non-critical decision. They are usuallyclassified in three levels:

- Department, if an error in the model would only affectone department or area.

- Institution, if an error would affect severaldepartments of an institution.

- External, if the error may affect the institution’sreporting to third parties such as the supervisor, therating agencies, the shareholders, etc.

Each model’s tier needs to be duly justified by the model’sowner and approved by the MRM function.

Model Documentation

The third key element in the development of models is theirdocumentation, on which there is increasing pressure fromsupervisors. Documentation is an effective lever for model riskcontrol when it is well managed.

It is necessary to complete the model’s documentation in acomprehensive manner before validation can proceed. Thedocumentation should follow unified templates approved bythe MRM function, and it should also ensure the entire modelcan be replicated by an independent third party, or eventransferred to a new modeler for it to be updated or improvedwithout this being a costly process.

A model’s documentation should include at least the following:

4 Data sources: databases used, extraction criteria applied,validations performed, staff responsible for data source andextraction, etc.

4 Model methodology report: model description, need andobjectives, reach, intended uses, limitations andassumptions, justification and detailed description of themethodology used, detail of the data used and justificationin terms of suitability, quality and robustness, etc.

4 Model calibration report: in the case of models withparameters calibrated with market or historical data, adetailed description of the methodology, appliedbenchmarks, quantification of estimator uncertainty, andfrequency, alerts and procedure for recalibration.

4 Test plan: description of the test plan the model hasfollowed during construction, together with the detailedresults.

4 User’s manual: in the case of models that are directlyexecuted by users, detailed instructions for applying themodel, limits and assumptions, guide for the interpretationof results and input data limits outside which the modelcannot work properly.

4 Technological environment and operational risk:description of the environment where the model isimplemented, assessment of the operational risk it involves(especially in cases of implementation outside the entity’stechnological environment) and detail of the contingencyplans foreseen in the case of operational failure.

23

Summing up, documenting models requires a substantial effortby the entity but is key to facilitating the update, follow-up,validation and audit processes, as well as the supervisoryreview. Consequently, it is receiving increasingly moreattention from entities and regulators.

Model Follow-up

Finally, it is essential to have a decision-model performancemonitoring system in place that allows for the early detectionof deviations from target and for remedial or preventive actionto be taken.

The follow-up should be carried out with a frequency that isproportional to the model risk (measured through its tier), andshould consist of a series of alerts and objective criteria that willmake it possible to determine when a model needs to berebuilt, recalibrated or decommissioned. Also, the final user’sfeedback is essential in model monitoring, as it is the mosteffective source for identifying deviations with respect to themodel’s expected behavior.

Model monitoring should be comprehensive, in the sense thatit should not be limited to the statistical or mathematicalalgorithm, but should instead monitor all elements employedin the decision (which can include decision rules, expertadjustments, strategies and any element that participates inthe final decision) jointly. This is especially relevant and a usualsource of model risk, because it is the model as a whole thatmakes the final decision.

Therefore, a comprehensive follow-up may examine thefollowing elements, among others:

4 Statistical model: the performance of the statisticalalgorithm is analyzed through:

- Analysis of sample stability through the evolution ofvariables.

- Metrics on predictive power development, such as thearea under the ROC curve, the powerstat or theKolmogorov-Smirnov distance, among others.

- Assessment of model behavior using techniques such asvolatility or predictivity analysis of each variable,backtest of obtained vs. expected results or analysis ofresiduals.

- Comparison of results with alternative models orindustry benchmarks.

4 Decision strategies: the behavior of decision rulesaccompanying the statistical model is analyzed; forinstance, minimum conditions for acceptance, exclusiveand restrictive rules or the cut-off point.

4 Expert adjustments: the impact of expert adjustments tothe model is analyzed; for instance, the manual decisionwhen the automatic judgment assigned by the decisionstrategy is modified (override).

For example, sound monitoring of a decision model in creditrisk would enable the assessment of many aspects, such as themodel’s predictive power, the cut-off point efficiency in termsof default, the binding character of the automatic decisionstrategy (scoring + rules), the proper functioning andimplementation order of the decision rules, the percentage ofmanual decision and, accordingly, the adequacy of the decisionauthority, override and the classification of their reasons, theopportunity cost (denied or withdrawn requests), or the use ofother criteria besides statistical criteria (e.g. profitability andcosts) in decisions, among others.

MANAGEM

ENT SO

LUTIONS

Model Risk Man

agem



24

To this end, it is useful to have tools that cover the entire modelfollow-up cycle (Fig. 9) and that make it possible to conduct awhat-if analysis and to quickly deploy to production all modelchanges arising from the follow-up observations27.

Model validation is a key element in model risk mitigation andits purpose is to effectively and independently challenge thedecisions made during model development, follow-up and use.

The frequency and intensity of the validation should beproportionate to the risk each model represents, measuredthrough its tier. Consequently, models assigned the highest risk(tier 1) must be validated by analysts that are sufficientlyexperienced and qualified and as frequently as their userequires, and their documentation should be especiallythorough, whereas lower risk models may be reviewed once ayear and the documentation requirements are much less strict.

According to the regulations and to best practice, modelvalidation should comply with a number of principles:

1. Thoroughness: all models that may involve risks for theinstitution should undergo the validation process.

2. Scope: the validation should not focus solely on themodels’ quantitative attributes, but it should also cover atleast the following aspects: - Methodology.- Documentation.- Quality of the data used.- Quantitative aspects.- Qualitative aspects (use tests, role of SeniorManagement, internal controls, etc.).- Technological environment.

3. Headcount and qualification: the Validation function shouldhave a sufficient number of qualified professionals for thedifferent aspects to be analyzed.

4. Independence: the institution should ensure that theValidation function is in a position to issue an opinion withfull independence, preventing any possible undueinfluence from units participating in the development ofmodels or other types of influence.

5. Responsibility: the institution is responsible for thevalidation process and cannot delegate it to third parties orthe supervisor.

6. Frequency: the validation is an iterative process that mustbe performed with a specific frequency (depending on themodel tier).

7. Internal criteria: there is no single standardized validationmethod for all institutions and models; each institutionneeds to set standards using its own criteria, and thesestandards should be commensurate with model risk.

8. Organization: the Validation function roles, responsibilities,work scheme and setting within the organization should bedocumented and approved at the corresponding level.

Fig. 9. Policies and Models Manager

27In this respect, Management Solutions has the MIR Policies and ModelsManager, designed with this architecture and functionality.

25

9. Documentation: descriptive information on differentaspects should be kept updated by the Validation function:

- Assessment, measurement and follow-upmethodologies, as well as statistical modelmethodologies.

- Validation reports, including those written by InternalAudit and those related to the validation process, with aclearly indicated conclusion (approved, approved oncondition - waiver - or disapproved.

- Historical record of the changes done in internalsystems, including the validation system itself.

10. Audit: the Validation function itself must be reviewed byInternal Audit, which needs to analyze its work andimplemented controls, as well as to give its opinion on thedegree of actual independence of this unit.

11. Vendor models: models developed by third parties (vendormodels) present additional difficulties for their validation,as the documentation may not be complete and theconstruction data may not be available. In these cases, theentity should request all necessary information and applythe same procedures to validate the model, and should doso with the same rigor as if it was internal, limited only bylegal requirements.

As was previously mentioned, while model risk quantification isnot required by regulations, in practice some entities arealready including specific quantitative techniques in their MRMframework for model risk mitigation purposes.

These techniques are applied to:

4 Data, through sensitivity to error in variables and even tothe absence of key data input for model execution.

4 Estimations, by using output sensitivity to the volatility ofestimators.

4 Use, by monitoring changes in predictive power and otherfollow-up metrics.

The following section explains how this quantification may beperformed, together with an exercise that illustrates itspractical application.

MANAGEM

ENT SO

LUTIONS

Model Risk Man

agem



26


27

Model risk quantitative management cycle

Beyond specific aspects required by the EBA for the valuationof derivatives (model risk AVAs28) , as was previouslymentioned, it is a fact that model risk quantification is not yetexplicitly required by regulators, which is why no significantprogress has been made by the institutions in this field so far29.

Notwithstanding the above and, for management purposes,some institutions have started to work on the model riskmanagement cycle from a quantitative perspective, with anaim to support qualitative model risk management asrecommended by the Fed and the OCC regulations. Seen inthis light, a model risk quantitative management cycle wouldcomprise three phases (Fig. 10):

4 Identification of model risk sources and classificationaccording to the previously mentioned sections:

- Data deficiencies.- Estimation uncertainty or model errors- Model misuse.

4 Quantification of the model risk inherent to each source,using a methodology based on model output sensitivity topotential fluctuations (in the inputs or the estimators) thatcharacterize the uncertainty associated to the source.

4 Mitigation of model risk identified and quantified byapplying the appropriate measures, which will depend onthe nature of each source.

Even though eliminating model risk is not possible,implementing an approach that combines a rigorous riskmanagement structure, as prescribed by the Fed and the OCC,with prudent and detailed quantification such as that describedmay be an effective strategy to mitigate it.

Motivation and approach of the study

In light of the scant practice observed in model riskquantification in the industry, only undertaken by someinstitutions, it was considered of interest to carry out anexercise in this connection. For this reason, a study wasdesigned with a view to quantifying this risk in credit riskparameters and models.

This exercise is composed of three parts, which coincide withthe three model risk sources described:

4 Data deficiencies: in this first phase, the impact of the lackof information in a model’s most predictive variables will be

Fig. 10. Model risk quantitative management cycle

28Model risk additional valuation adjustments (AVAs), detailed in EBA (2013).29The capital buffer is an exception in the CCAR exercise in the US, which someentities are implementing.

MANAGEM

ENT SO

LUTIONS

Model Risk Man

agem



28

analyzed; for example, a scoring model for consumermortgages will be used. To do this, we will reconstruct thescoring model without these variables, and determine theassociated decrease in predictive power and its relationshipwith the model management outputs: the assumed default(type I error) and the missed potential business oropportunity cost (type II error).

4 Estimation uncertainty or model error: in the second part,the model risk arising from uncertainties in the estimationwill be studied. To this end, the confidence intervals of theestimators obtained from the scoring model, the PDcalculation in the calibration and the LGD calculation willbe used to describe the normal distributions thatcharacterize these estimators. Various parameter sets willbe simulated on these distributions using Monte Carlo andregulatory capital for the portfolio will be re-estimatedbased on the outcome. This will provide a distribution ofthe capital requirement for the portfolio, its volatility arisingexclusively from the uncertainty in the selection ofestimators.

4 Model misuse: finally, the risk arising from a scoring modelthat was not properly monitored and updated to keep upwith portfolio evolution over time will be analyzed. To dothis, the model’s predictive power at the time ofconstruction will be compared with that shown a year later,and the effect of not updating the model on type I and IIerrors will be analyzed.

With this, three real examples of model risk quantification willhave been provided to offer a measure of its relevance.

Data

The study was carried out using the following data, models andparameters30:

4 A real construction sample for a mortgage scoring model.The sample comprises some 200,000 mortgage loans, witha default rate of around 4%.

4 A real implementation sample, a year after construction, ofthe mortgage loan portfolio at the time of its constructionand its performance in the following year.

4 A mortgage scoring model built on the previous sample. Ithas 14 variables and its predictive power is medium-high(ROC31 of around 78%).

4 Calibration mechanisms for the PD, LGD and CCFparameters of the mortgage portfolio studied.

4 The BIS regulatory capital calculation mechanism by the IRBmethod.

Main findings

The main findings from this study are the following:

4 As regards the model risk arising from the data, it isobserved that the presence of errors in the three mostpredictive variables of a scoring model may double thedefault rate on the balance sheet, or may instead reduce by40% the business obtained if the same default rate is to bekept.

4 Quantifying the impact of model estimation uncertaintyreveals that the capital requirement for a mortgageportfolio may be underestimated by up to 8%, with aconfidence level of 90%, due to the combined effect ofuncertainty in the score estimators (4% underestimation ifconsidered in isolation), in PD calibration (7%) and in LGDestimation (2%).

4 Finally, with regard to model misuse, it is observed thatfailure to update a model for 12 months may cause itspredictive power to decrease by around 10%. This in turnresults in an increase of up to 67% in the default rate, oralternatively, a 15% reduction in the volume of acquiredbusiness (i.e. a 15% increase in the opportunity cost) if thecut-off point is fixed so as to maintain the same defaultrate.

Summing up, in addition to the previously describedqualitative effects, model risk may have substantial quantitativeimpacts that could lead to erroneous management decisions orto an institution’s capital requirement to be underestimated.Consequently, some banks include robust model riskquantification techniques in their MRM frameworks in order toproperly mitigate this risk.

30All data, models and parameters have been partially modified to ensureconfidentiality and at the same time maintain the study’s robustness andrepresentativeness. 31Receiver operating characteristic: measure of the predictive power of a binaryresponse model.

The estimation of sample parameters is normally accompanied bythe obtainment of confidence intervals, which provide a morerealistic estimation of the parameter since they represent a range orset of values within which the real value of the estimator lies witha given probability (established by the level of confidence). As aresult, an interval with a level of confidence 1 – α for a parameter 𝜗is represented as:[�₁,��] where 𝑃(�₁��) = 1-𝛼There are two general approaches for the estimation of confidenceintervals:

- Parametric, in which a known distribution of the estimator isassumed.

- Non parametric, in which no knowledge about the distributionis assumed.

Below are several common techniques for the obtainment ofconfidence intervals of both parametric assumptions (taking alogistic regression as an example) and non-parametric assumptions(such as the calculation of central-tendency estimators, such as themean or the median).

Confidence intervals of the logistic regression estimators(parametrical methods)

Generally, logistic regression is used to relate discrete binaryresponses (such as the non-payment or payment of a debt) with aset of explanatory variables, which is the reason why this is themost frequently used algorithm in scoring and rating models. Thisrelationship may be expressed as follows: 𝑙𝑜𝑔𝑖𝑡 (𝜋) = 𝑙𝑜𝑔 � � = 𝛼+𝛽´𝑥Where 𝑥 is the vector of explanatory variables used in the model, 𝜋 is the probability that the 𝑌 response being predicted actuallyhappens, conditioned to 𝑥, that is, 𝜋=𝑃(𝑌=1|𝑥), 𝛼 is theintersection at the origin of the curve, and 𝛽 is the vector with theslopes of the 𝑥 variables.

When modeling through a logistic regression, the 𝛼 and 𝛽parameters, known as weights or estimators, are obtained. Thus,the main source of model risk in a logistic regression is the errormade when estimating its weights, and confidence intervals maybe used to quantify it.

Wald’s confidence intervals, also called normal confidenceintervals, are based on the fact that estimators show a normalasymptotical behavior, and admit a closed expression. As a result,Wald’s confidence interval 100�(1-𝛼)% for the 𝛽� estimator issimply: 𝐼� = 𝛽�� ± 𝑧�-�/�𝜎̂�where 𝑧� is the 100�𝑝% percentile of the standard normaldistribution, 𝛽�� is the maximum likelihood estimator of 𝛽�, and 𝜎�̂is the standard error estimation of 𝛽��.From the confidence intervals obtained, it is easy to get theempirical distribution of the score for each of the logs in thesample, which is the key element to calculate the model error thatis being assumed. The steps to do this would be as follows:

- For each 𝛽� estimator, the function of the normal distribution 𝐹�=𝑁�𝛽��, 𝜎�̂� is considered, centered in the weight estimator andwith standard deviation equal to the estimator of its standarderror.

29

- A large amount (𝑛) of random numbers 𝑥 is simulated,according to an even distribution between 0 and 1, 𝑋~𝑈(0,1).

- For each 𝑘 ∈ {1..𝑛}, a full set of 𝛽� estimators for the logisticregression is simulated through the inverse of its respectivedistribution functions, 𝛽�� = 𝐹�–1 �𝑥��.

- The entire portfolio is scored with each set of estimators.

This way, the distribution of the score for each portfolio record isobtained, from which it is possible to estimate confidence intervals,volatilities and other measures for the quantification of the modelrisk shown by the natural uncertainty in the estimation of a logisticregression.

Confidence intervals of central tendency estimators (non-parametric methods)

To establish the uncertainty associated to an estimator that hasbeen calculated using a central tendency estimator (such as a meanor a median, used in the calculation of LGD in credit risk, forexample), non-parametric techniques that do not require knowingor assuming a distribution for the estimator are normally used. Oneof the most common techniques is bootstrapping, which is based onrandom sampling and basically consists of generating n sizesamples to obtain the distribution function of the selectedestimators (mean, median, etc.) from all the generated samples.

In particular, the steps to develop the confidence intervals of theselected estimator for the calculation of a 𝛽 estimator would be thefollowing:

- Given the parameter construction sample composed of 𝑛observations, 𝑥 = (𝑥₁, … 𝑥�), a random sample is generated toreplace the original sample to obtain a bootstrap sample 𝑥* = (𝑥 = (𝑥₁, … 𝑥�).

- Then, the statistic of interest for the bootstrap sample 𝛽* = 𝛽(𝑥*)is generated.

- Steps 1 and 2 are repeated 𝑁 times being 𝑁 sufficiently large.Generally, it is suggested that 𝑁 reaches values over 1,000iterations32 for the calculation of intervals with confidence levelsbetween 90% and 95%.

- After the 𝑁 iterations, a bootstrap estimator sequence 𝛽*₁, …, 𝛽*� will be generated for the statistic distribution to bestudied.

The distribution of bootstrap estimators will enable the calculationof a percentile confidence interval for 𝛽 (1–𝛼)�100%, among otherthings. This interval will be given by the quantiles (𝑞�(𝑝₁), 𝑞�(𝑝�)) of𝑁 bootstrap replications with 𝑝₁=𝛼/2 and 𝑝�=(1–𝛼)/2.

32Efron and Tibshirani (1993).

Quantification of uncertainty: confidence intervals

𝜋1-𝜋

MANAGEM

ENT SO

LUTIONS

Model Risk Man

agem



30

Part I: Data deficiencies

The first exercise focuses on quantifying the impact of modelrisk through data deficiencies (Fig. 11). To do this, two modelsare developed from a real mortgage scoring model:

4 Model A: the score resulting from removing the mostpredictive variable and retraining the model.

4 Model B: the score resulting from removing the three mostpredictive variables and retraining the model.

In the first case, a decrease in predictive power of almost 2percentage points is observed, and of nearly 9 percentagepoints in the second case.

From the two obtained models, the entire portfolio is rescoredand two what-if analyses are performed on each model,moving the cut-off point in order to answer the followingquestions:

4 If a cut-off point were fixed in the new model so as to keepconstant the default accepted by the model (type I error),how much would the opportunity cost increase, measuredas the potential business volume that is not obtained?

4 If, conversely, a cut-off point were fixed in the new modelso as to keep constant the business volume that the modelaccepts (type II error), how much would the defaultaccepted by the model increase?

The results obtained are as follows (Fig. 12):

4 When reconstructing the model without the mostpredictive variable, keeping the default rate equal to that ofthe original model reduces the business volume obtainedby 5% (i.e. the opportunity cost is multiplied by 1.05), and

keeping the business model obtained by the original modelalso multiplies the default rate by 1.05.

4 When reconstructing the model without the three mostpredictive variables, keeping the default rate equal to thatof the original model reduces the business volumeobtained by 40% (i.e. the opportunity cost is multiplied by1.40), and keeping the business volume obtained by theoriginal model also multiplies the default rate by 1.98 (i.e. itis almost doubled).

As can be seen, the presence of errors in the most predictivevariables -if they are sufficient to invalidate their use in themodel- has very significant effects on the default and businessvolume that the model approves.

Fig. 12. First exercise results: Change in the default rate vs. Opportunitycost

Fig. 11. Summarized approach of the first exercise

MANAGEM

ENT SO

LUTIONS

Model Risk Man

agem



32

Part II: Uncertainty in the estimations

The second exercise seeks to quantify the impact of model riskthrough the implicit uncertainty in estimators (Fig. 13). To thisend, based on a real scoring model, the PD calibration, the LGDestimation and the BIS capital formula, four Monte Carlosimulations are performed:

4 From the confidence interval for each scoring modelestimator, their normal distributions are reconstructed.Using these normal distributions, 10,000 parameter sets aresimulated, therefore 10,000 models, which are used to ratethe entire portfolio and to calculate capital, keepingconstant the original PDs and LGDs. This way, the capital’ssensitivity to uncertainty in the scoring model estimators iscalculated.

4 From the confidence interval of the PDs in the calibration,10,000 PD sets are simulated, and with them the capital iscalculated, keeping the original scoring and LGD constant.This way, the capital’s sensitivity to uncertainty in the PDcalibration is estimated.

4 Similarly, a simulation on the LGDs is performed, keepingthe original score and PD constant, to determine thecapital’s sensitivity to uncertainty in the LGD estimation.

4 Finally, an aggregated simulation is carried out, in whichboth the scoring estimators and the PDs and LGDs fluctuateaccording to their confidence intervals, and capital iscalculated. With this, the sensitivity of capital to thecombined uncertainty from the score, PD calibration andLGD estimation is calculated.

With a level of confidence of 90%, the findings are as follows(Fig. 14):

4 As a result of the uncertainty in the score estimators, thecapital requirement could move up or down by 4%.

4 The uncertainty in PD calibration could cause the capitalfigure to move up or down by 7%.

4 The uncertainty in LGD estimation could cause capital tomove up or down by 2%.

4 Finally, the combined effect of the three previoussimulations shows that capital could move up or down by8% due to the model risk arising from the uncertainty in itselements.

As can be observed, the uncertainty in the estimations couldcause a situation in which capital is underestimated by up to8%, due solely to model risk, a fact that should be consideredand mitigated -if possible- with a conservative approach.

Fig. 13. Summarized approach of the second exercise

Fig. 14. Second exercise results: capital variation (multiplier) for eachsimulation and for the combined simulation

33

- If, conversely, when applying the model at the end ofthe 12th month, a cut-off point were fixed so as to keepunchanged the business volume obtained by the modelat construction (type II error), how much would thedefault accepted by the model increase?

The findings are as follows (Fig. 16):

4 A decline in predictive power of over 10% (8 ROC points) isobserved after 12 months.

4 When the model is applied 12 months after construction,maintaining the default rate as it was at the time ofconstruction reduces the business volume obtained by 15%(i.e. the opportunity cost is multiplied by 1.15), whereasmaintaining the business volume obtained by the originalmodel multiplies the default rate by 1.67.

Therefore, the third exercise shows that the model risk arisingfrom misuse (in this case, not having updated it) may have asignificant impact on default and on the volume of acquiredbusiness.

Part III: Model misuse

The last exercise seeks to quantify the impact of model riskarising from misuse (Fig. 15). Misuse may include applying amodel to a population other than the one used to constructthe model (for example, applying a scoring model built on thebasis of a mortgage portfolio from a specific country to thesame portfolio from another country, or to the portfolioresulting from a merger with another entity). A special and notso evident case of inappropriate use would be applying amodel to the same portfolio after a long period of time withoutverifying that the model is still applicable, especially if there hasbeen an economic cycle change that could have substantiallytransformed the population.

To reflect this case, the following exercise is based on a scoringmodel, and seeks to quantify the impact of not having updatedit for 12 months. To do this:

4 The model is applied to the portfolio at the end of each ofthe 12 months following its construction.

4 The actual default levels shown in the following year areused to measure the model’s predictive power, whichappears to decrease over time.

4 Two what-if analyses are performed by moving the cut-offpoint to answer to the same questions as in the firstexercise:

- If, when applying the model at the end of the 12thmonth, a cut-off point were fixed so as to keep constantthe default rate that the model accepted at construction(type I error), how much would the opportunity costincrease, measured as the business volume that is notobtained?

Fig. 15. Summarized approach of the third exercise

Fig. 16. Third exercise results: What-if analysis of Default vs.Opportunity cost

MANAGEM

ENT SO

LUTIONS

Model Risk Man

agem



34

Model risk mitigation

As can be seen from the study, quantifying model risk directlyis a complex task, and quantitative measures to cover this risk,such as a capital buffer or a provision for loan losses, are notalways applicable or make any sense.

In practice, this is why institutions focus their model riskquantification efforts on identifying model sensitivity to errorsthat could result in losses (whether economic or reputational).

The quantification cycle is therefore completed by the possiblemitigating actions on model risk that are drawn from thissensitivity analysis. These actions may include the following(according to each model risk source):

4 Strengthening of data governance.

4 Establishment of a Data Quality function to ensure thequality, integrity, traceability and consistence of the datathat feeds the models.

4 Introduction of a set of input validation rules and of expertrules on outputs, especially as regards data deficiencies.

4 Qualification and experience of the model’s validators andmodelers.

4 Effective, critical and independent validation of models.

4 Measured and duly justified conservatism in inputs,estimations and outputs.

4 Periodic model backtesting to compare expected outputswith actual outputs and conclude on the degree of modelaccuracy.

4 Stress test of the model, putting the model inputs throughdifferent stress scenarios and concluding on the model’sperformance in such situations.

4 Academic support or market benchmark, when applicable,for the methodological decisions adopted.

4 Use of alternative models to contrast results.

4 Development of complementary analyses that question thevalidity of the model results using additional information.

4 Occasionally, provision of a capital buffer or an allowancefor model risk losses.

4 Rigorous governance of models to include setting limits ontheir use and expressly validating and approving them foreach use.

4 Greater human supervision, especially during the firstphase after deploying the model to production.

4 Regular model follow-up, including the frequent andautomated monitoring of the model’s predictive power andan early warning system for deterioration.

4 Thorough inventory of the institutions’s models, with theobligation to register every model used in decision making.

4 Pilot testing before initial deployment to production andafter every substantial change for high risk models.

The propagation of uncertainty is an unavoidable effect ofstatistical models that arises when a model is fed with theestimation provided by another model, which can amplify theerror.

For example, the result of credit scoring models is an inputvariable for the calibration of the probability of default (PD), whichin turn feeds capital consumption models. The combined statisticaluncertainty at each step may cause high volatility in the final result(the capital, in this case) due to model risk.

In statistics, the most common method for measuring uncertaintyis the use of the absolute error of variables ∆𝑥 and of theirstandard deviation, 𝜎. If variables are correlated, their covariancemust also be considered for the calculation of the errorpropagation.

Propagation of error in linear combinations

Let ƒ�(𝑥₁, ... ,𝑥�) be a set of 𝑚 functions which are linearcombinations of 𝑛 variables 𝑥₁,... ,𝑥� with combination coefficients𝐴�₁, ... ,𝐴��, (𝑘=1...𝑚); that is:

ƒ� = 𝐴��𝑥� = 𝐴𝑥Then, the variance-covariance matrix of ƒ is given by:

𝑐𝑜𝑣��= 𝐴��𝑐𝑜𝑣��𝐴�� = 𝐴𝑐𝑜𝑣�𝐴�Where 𝑐𝑜𝑣� is the variance-covariance matrix of the set of 𝑥variables.

The previous expressions represent the error propagation of a setof variables in the function to which they belong. When the errorson the set of variables 𝑥 are uncorrelated, the previous expressionsimplifies to: 𝑐𝑜𝑣��= 𝐴��(𝜎2�)� 𝐴��Propagation of error in non-linear combinations

When ƒ is a non-linear combination of the set of variables 𝑥, aninterval propagation could be performed in order to computeintervals which contain all consistent values for the variables. In aprobabilistic approach, the function ƒ must be linearized byapproximation to its first-order Taylor series:

ƒ� ≈ƒ0� + 𝑥�≈ƒ0 + 𝐽𝑥where 𝜕ƒ��𝜕𝑥� denotes the partial derivative of ƒ� with respect tothe 𝑖-th variable and where 𝐽 is the Jacobian matrix of ƒ. Since ƒ⁰ isa constant, it does not contribute to the error on ƒ. Therefore, thepropagation of error follows the linear case above, but replacingthe linear coefficients,𝐴�� and 𝐴�� by the partial derivatives 𝜕ƒ��𝜕𝑥�and 𝜕ƒ��𝜕𝑥�: 𝑐𝑜𝑣�= 𝐽𝑐𝑜𝑣(𝑥)𝐽�

Examples

This table shows the variances of different functions of the realvariables A, B with standard deviations 𝜎A,𝜎B, covariance andconstants a,b. Since most statistical models can be constructed as acomposition of these functions, inferring from them how thevolatility of the original variables is transferred to the final result ofthe combined model is immediate.

As can be seen, the error volatility can quickly increase just byconcatenating two statistical models, which has seriousimplications when quantifying model risk in a decision system andreinforces the need to establish reasonableness controls in theintermediate results.

35

Propagation of uncertainty in model concatenation

��₌₁ � ��

� � �

� � � 𝜕ƒ�𝜕𝑥�

MANAGEM

ENT SO

LUTIONS

Model Risk Man

agem



36

Bank of Spain (2007). Validation Document no. 2. Criteria forinternal validation of advanced risk management models

Bank of Spain (2008-14). Guidelines on the Internal CapitalAdequacy Assessment Process (ICAAP) at credit institutions (2009,2011 and 2014 reviews)

Bank for International Settlements (2013), Statistical release OTCderivatives statistics at end-June 2013.

Basel Committee on Banking Supervision (2004-06). InternationalConvergence of Capital Measurement and Capital Standards: ARevised Framework - Comprehensive Version (2006, june).

Basel Committee on Banking Supervision (2005-1). BaselCommittee Newsletter No. 4 (January 2005). Update on work of theAccord Implementation Group related to validation under theBasel II Framework.

Basel Committee on Banking Supervision (2005-2). BaselCommittee Newsletter No. 6 (September 2005). Validation of low-default portfolios in the Basel II Framework.

Basel Committee on Banking Supervision (2006). Basel CommitteeNewsletter No. 9 (September 2006). The IRB Use Test: Backgroundand Implementation.

Basel Committee on Banking Supervision (2010-11). Basel III: Aglobal regulatory framework for more resilient banks and bankingsystems (2010, December; revised on 2011, June).

EBA-CEBS (2006). Guidelines on the implementation, validation andassessment of Advanced Measurement (AMA) and Internal RatingsBased (IRB) Approaches.

EBA (2013). RTS on Prudent Valuation, Consultation Paper.

EBA (2014). Model Validation (Single Rulebook).https://www.eba.europa.eu/regulation-and-policy/model-validation (in the pipeline).

Efron, B. y Tibshirani, R. J. (1993). An Introduction to the Bootstrap.Chapman and Hall, London.

Management Solutions (2013). Impact analysis of stress tests in thefinancial system.

MIT Sloan Management Review. T. H. Davenport y J. G. Harris(2005). Automated Decision Making Comes of Age.

Office of the Comptroller of the Currency and Board of Governorsof the Federal Reserve System (2011-12). Supervisory Guidance onModel Risk Management.

Securities & Exchange Commission (SEC) and Commodity FuturesTrading Commission (CFTC) (2010). Findings regarding the marketevents of May 6, 2010. Report of the staffs of the CFTC and SEC tothe Joint Advisory Committee on Emerging Regulatory Issues.

Bibliography

37

Glossary

ALM: Assets and liabilities management.

Capital buffer: Capital surcharge, whose purpose is to ensure thatan entity is able to absorb the losses derived from its activity inperiods of stress.

CCF: Credit conversion factor.

EAD: Exposure at default. It has an ‘off balance sheet’ component(commitments, etc.) which requires certain assumptions to bemade. For instance, it could be said that for a credit facility, EAD isequal to the amount drawn down + CCF x available credit amount,where CCF is the credit conversion factor.

EBA (European Banking Authority): an independent EU authoritywhose overall objective is to maintain financial stability in the EUand to safeguard the integrity, efficiency and functioning of thebanking sector. The EBA was established on 1 January 2011 as partof the European System of Financial Supervision (ESFS) and tookover the Committee of European Banking Supervisors (CEBS).

Fed (Federal Reserve System): central bank of the United Statesfounded in 1913 to provide the nation with a safer, more flexibleand more stable monetary and financial system. Over the years, itsrole in banking and the economy has expanded to includeactivities such as conducting the nation's monetary policy,supervising and regulating the banking institutions or providingfinancial services to depository institutions.

Flash crash: a very rapid, deep, and volatile fall in security pricesoccurring within an extremely short period of time. For example,the flash crash occurred on 6 May 2010 in the United States.

Gini index or powerstat: metric that serves to quantitatively analyzethe discriminant power of a binary output model, based on theclassification it makes of adverse and favorable events.

IRB (Internal Rating Based): advanced method for estimatingregulatory capital based on internal rating models. To gain access,institutions must fulfill a number of requirements and beauthorized by the supervisory authority.

Kolmogorov-Smirnov distance: non-parametric test used tocompare the similarity between two probability distributions. Ituses the maximum of the absolute difference between theempirical and the estimated distribution. It is used as a predictivepower metric in binary output models.

KYC (know your customer): relevant information on customersobtained for various purposes, such as regulatory compliance onfraud, money laundering, terrorist financing or corruption.

LGD: Loss given default. It is equal to 1 – the recovery rate.According to BIS II, paragraph 468, it must be calculatedconsidering economic downturn conditions.

LTV: relationship between the outstanding amount of a loan andthe value of the associated collateral (loan to value). Used forcollateralized loans, mainly mortgages.

Model Risk Additional Value Adjustment (AVA): necessaryadjustment for the proper valuation of positions in the tradingbook.

Monte Carlo simulation: technique used to approximate theprobability of an event by carrying out multiple simulations usingrandom variables.

OCC (Office of the Comptroller of the Currency): US federal agencywhose mission is to regulate and supervise all national banks,federal bureaus and foreign bank agencies. The OCC’s goal is toensure that the supervised organizations operate in a safe andsound manner and in compliance with legal requirements,including the fair treatment of customers and their access to thefinancial market.

Override: manual decision that contradicts the result of a statisticalmodel.

PD: probability of default.

ROC curve (receiver operating characteristic): curve used to analyzethe predictive power of a binary output model. It represents therelationship between the type I error (incorrectly classifyingadverse events) and the type II error (incorrectly classifyingfavorable events).

RWA: risk weighted assets; on or off balance sheet exposureweighted by the risk involved for the institution, calculatedaccording to the methods established by the regulator.

Scoring/rating: model that assigns a score to each rated item(facilities/counterparties) according to their credit quality. If what isbeing rated is a facility-customer binomial, it is a scoring model;but if counterparties are to be rated, it is a rating model.

Stress test: simulation technique designed to determine the abilityof an entity to withstand an adverse financial situation. In a broadersense, it refers to any technique used to assess the ability to dealwith extreme conditions, and it can be applied to entities,portfolios, models, etc.

Type I error: statistical term referring to the error that occurs whenthe null hypothesis is true, but it is rejected.

Type II error: statistical term referring to the error that occurs whenthe null hypothesis is accepted even though it is false.

VaR (Value at Risk): statistical technique used to quantify the levelof financial risk assumed over a period of time at a givenconfidence level.

What-if analysis: simulation of the impact of one or more specificscenarios for the input variables on the outputs of a process.

MANAGEM

ENT SO

LUTIONS

Model Risk Man

agem



38

Management Solutions is an international consulting servicescompany focused on consulting for business, risks, organizationand processes, in both their functional components and in theimplementation of their related technologies.

With its multi-disciplinary team (functional, mathematicians,technicians, etc.) of over 1,300 professionals, ManagementSolutions operates through its 18 offices (9 in Europe, 8 in theAmericas and 1 in Asia).

To cover its clients' needs, Management Solutions has structuredits practices by sectors (Financial Institutions, Energy andTelecommunications) and by lines of activity (FCRC, RBC, NT),covering a broad range of skills -Strategy, CommercialManagement and Marketing, Organization and Processes, RiskManagement and Control, Management and FinancialInformation, and Applied Technologies.

In the financial sector, Management Solutions offers its servicesto all kinds of companies -banks, insurance companies,investment firms, financial companies, etc.- encompassing globalorganizations as well as local entities and public bodies.

Luis Lamas Management Solutions Partner [email protected]

Carlos F. GallejonesManagement Solutions [email protected]

Ruben G. MoralManagement Solutions Director [email protected]

Juan G. CascalesR&D Manager at Management [email protected]

Javier Calvo R&D Manager at Management [email protected]

Our aim is to exceed our clients'expectations, and become their trusted

partners

Design and LayoutMarketing and Communication DepartmentManagement Solutions - Spain

© Management Solutions. 2014All rights reserved

Madrid Barcelona Bilbao London Frankfurt Warszawa Zürich Milano Lisboa BeijingNew York San Juan de Puerto Rico México D.F. Bogotá São Paulo Lima Santiago de Chile Buenos Aires

Model Risk Management: Quantitative and qualitative aspects

Documents

Transcript of Model Risk Management: Quantitative and qualitative aspects