www.fico.com Make every decision countTM

INSIGHTS WHITE PAPER

Number 82

Reducing Regulatory Drag on Analytics Teams Automated workflows standardize and speed model management processes

Banking regulators are increasing scrutiny of analytic models, peeling back layers of the onion with

probing questions. They want to know not only how models affect credit policies and customer

decisions, but about the processes used for developing, validating, deploying and updating them.

Banking executives, increasingly aware of the full dimensions of model risk, are also asking pointed

questions.

Finding answers can add drag to the performance of analytics teams—even pulling them away from

high-value work that leads to competitive advantage.

To improve compliance and response time to detailed questions, leading banks are implementing

formal model management processes throughout the analytic lifecycle. But while best practices may

be understood, they can be challenging to deploy consistently

across analytic teams. It’s also difficult to know if they’re being

followed at the right level of granularity, such that no matter

where regulators probe—and even with analytic staff turnover—

all questions can be readily answered.

This white paper examines how automated, configurable model workflow tools promote process

consistency and accountability. We show how banks are using workflow at enterprise and

departmental levels to improve model governance without creating extra work for analytic teams.

In fact, by orchestrating model processes while automatically capturing key artifacts, decisions and

sign-offs, workflow can lead to better model performance. Analytic teams are freed to spend more

of their time creating and updating models.

We’ll cover:

• Turning work friction into work flow

• Creating standard processes that fit the needs of diverse teams

• Tracking model and characteristic lineages

Improve compliance while spending 75% less time on it.

Reducing Regulatory Drag on Analytics Teams

INSIGHTS WHITE PAPER

November 2014 www.fico.com page 2

Banks, long at the vanguard of data analytics for business, have continued to expand their use

of models. Predictive scorecards and other models for anticipating and responding to customer

behavior now play a central role in every area of operational decision making. The sheer quantity

of these models is increasing rapidly.

What hasn’t advanced as quickly is model management. An August 2014 article by

Butler Analytics pointed out that “some banks simply do not know how many models are

actually deployed.” In other cases, model information is in so many places that preparing for

audits or answering regulator inquiries becomes extremely labor-intensive. The Butler article

reports that “the modeling staff in one major US bank now spend 80% of their time meeting

regulatory requirements, detracting from much needed new model development.”

While this is an extreme case, every bank is seeing an impact on the productivity and

performance of its analytics teams. Both those building models and those compiling reports

and answering inquiries have more work to do. And where one group is charged with both

duties, compliance tasks can siphon away analytic resources from work that could be producing

significant value for the bank.

An executive in the mortgage division of one bank told FICO that, for a period of time, 25% of its

analytics workforce had to be diverted to collecting, preparing and reporting on data required by

regulators—costing the bank tens of millions of dollars.

Yet some banks are moving ahead of the curve. They’re improving their ability to answer

questions about analytics while lightening the burden on their analytics teams. In fact, a bank

devoting 80% of modeler time to regulatory requirements could reduce that expenditure to 20%

or less.

Model Management—Getting Ahead of the Curve

Over the past five years, analytic excellence has become a core requirement in today’s financial market… Modeling touches virtually every decision of the bank… IDC Financial Insights A Framework for Model Governance, June 2013

“Organizations today face heavy regulatory pressures…To meet these challenges and mitigate risk, they need model management solutions that can reduce resources required to complete compliance

audits, and encompass the full model lifecycle and risk-management continuum...” ” Peyman Mestchian Managing Partner at Chartis

www.fico.com page 3

Reducing Regulatory Drag on Analytics Teams

INSIGHTS WHITE PAPER

This more efficient, streamlined approach combines centralized model management with

automated, configurable workflow tools. As depicted in Figure 1, workflows enforce approved

processes at every stage of the model lifecycle. They capture granular information about what

decisions and actions are taken and why.

Populated by these workflows, the central repository maintains an inventory of all models in

operations and under development—their purpose, data types/sources, key assumptions,

exclusions, predictive characteristics,

segmentation schemes, where model is

being used, restrictions on usage, etc. The

model management solution also automates

scheduled validations and flags models for

review when stability or performance metrics

decline. This makes it much easier for banks to

maintain models at peak performance, as well

as spot and resolve any compliance issues.

Analytics teams accessing this shared resource

have detailed information at hand to explain,

for example, how they analyzed population

segments for a particular model—or across

multiple models—and defend their choice of

segmentation. They can readily justify why a

new predictive characteristic was added during

a model refresh and how it was developed.

They can provide regulators with evidence that

a rigorous analysis of all potential economic

drivers was performed for Long Run and

Downturn PD estimation.

Turning work friction into work flow

We’ve discussed how increasing regulatory

scrutiny has affected the work processes

of bank analytic teams. Now here’s the real

rub: This burden is settling on analytics

teams at exactly the moment when the

importance of what they do for helping banks

understand customers and create competitive

differentiation has become strikingly clear.

Banking executives want their analytic teams

to bring more powerful, complex analytics to

market as quickly as possible. But they also

need them to capture all of the details and

reasons along the way so that models and

their usage are transparent and explainable to

regulators, customers and executives.

FIGURE 1: BEING PREPARED TO ANSWER ANY AND ALL REGULATOR QUESTIONS

Banks can answer such questions with confidence when they have standardized, approved processes in place across the model lifecycle...

...and one place to access all process history details

Regulators are asking more detailed “peel back the onion” questions about models, such as:

When was this model redeveloped?

Why was this new characteristic added?

How was this characteristic calculated?

New customer characteristic:Ratio velocity retail purchases to velocity ATM withdrawals (Predictive of rising credit risk)

CEN

TRA

LIZE

DM

OD

EL M

AN

AGEM

ENT

Behavior Score Redevelopment Process History

VALIDATION

SUMMARY

PlansResultsMetadataVariables & RolesConfigurationProcessesModel Usage

DEFINITION

Models Alert Rulesets Validation Jobs Targets

WorkspaceDashboard AdministrationProcess AuthoringAnalytics Team A

Task Name AssigneeVariable Value

Task Activity

Independent review D. Berman IR_Approval_Status Approved

Develop model J. Kalabar Technical_Review Behavior Score Redevelopment Technical Review.doc

Data quality review R. Santos DQ_Status Approved

Assess data quality J. Kalabar DQ_Report Behavior Score Redevelopment Data Quality Report.docRole assignment L. Moreau Project_Team_Status Assigned

DQ_Comments Key fields look good. No problem that PromoCode is scarcely populated.

Model_Specs Behavior Score Redevelopment Specification.doc

IR_Comments Great work. Let’s proceed with implementation testing.

1

2

3

Model development documentation provides information on how the characteristic was developed as well as its predictive power.

UPDATED BEHAVIOR SCORE

Development Validations Redevelopment

BEHAVIOR SCORE LIFECYCLE

www.fico.com page 4

Reducing Regulatory Drag on Analytics Teams

INSIGHTS WHITE PAPER

Executives also want to get answers that allay their concerns about model risk. It’s not just

potential financial and reputational damage from regulatory noncompliance that’s worrisome,

but the opportunity costs of delayed model deployments and competitive impacts from

making decisions with underperforming models.

Automated workflow reduces the friction of these additional requirements for the entire

analytics team.

For model development: Workflow guides users through standard processes for

documenting actions and decisions at each step. (See Figure 2.) It eliminates manual

data input by automatically capturing segmentation criteria, test results and other key

information. It also orchestrates review cycles across departments, notifies stakeholders

and ensures approvals are obtained for moving on to the next step. Dashboards provide

at-a-glance status on the real-time progress of all models.

For regulatory and internal reporting and inquiries: Workflow feeds data about

model lifecycle processes into a centralized repository. Users compiling reports, preparing

for audits or responding to queries work faster and more efficiently with this single source

of all relevant information on models and modeling processes. They can very quickly, for

instance, examine a model’s most recent validation test. But they can also just as easily

peruse an entire process history encompassing every validation ever performed on that

model, and the results and consequent actions in each instance.

FIGURE 2: AUTOMATED WORKFLOW ORCHESTRATES MODEL LIFECYCLE PROCESSES WHILE CAPTURING DATA AND DOCUMENTATION FOR CENTRALIZED MANAGEMENT

Models Processes My TasksWorkspaceDashboard AdministrationProcess Authoring

Analytics Team A

Redevelop Behavior Model

Task: Data Quality Review

Your Actions

Using the form below, review the model data quality assessment and supporting documentation, and render an approval decision

Decision Action: Comments:

Supporting InformationModel Name:

Change Status:

Change Summary:

Model Owner:

Project Docments:

Comment History:

Behavior Score

Pending ReviewUpdating the model in response to validation

review recommendations and to incorporate

new transactional predictive characteristics

J. Kalabar

Behavior Score Redevelopment Techinical Review.docPlease note in the documentation which records were

used for the out-of-time validation of the new parameters

Approval Required:

Model review triggered by out-of-range results during periodic automated validation; recommenda-tion to redevelop

Process initiated by model manager

Email to analytics team leader: Tasks required: Project approval & role assignment

Email to model developer: Task required: Data quality assessment

Email to technical reviewer: Task required: Data quality approval

Data & documentationcaptured in shared repository

Development Validations Redevelopment

BEHAVIOR SCORE LIFECYCLE PROCESS MANAGEMENT

1

2 3 4

5

www.fico.com page 5

Reducing Regulatory Drag on Analytics Teams

INSIGHTS WHITE PAPER

Overall, this approach provides banks (and regulators) with

complete visibility into how each model is developed, deployed

and maintained. It ensures that the evidence banks need to

explain and defend their analytic choices is always fully captured

and readily accessible.

Creating standard processes that fit the needs of diverse teams

Banks vary greatly in how they use analytics across their

enterprise and the scope of their efforts to standardize lifecycle

model management processes. Here are some examples:

• One FICO client, a top-five US bank, is moving to centralize

management of every model across its vast enterprise.

• A leading Asian bank is initially focusing on ensuring its models

in development achieve Advanced Internal Rating Based status

under the Basel III global standard.

• A top-five Australian bank seeks to bridge current process

inconsistencies around model tracking and validation of Basel

rating models and decision models across different countries.

For any initiative, flexibility to align workflows with the needs

of analytic teams is essential. Managing best practices depends

partly on the types of analytics that teams are developing. For

instance, predictive models for forecasting customer behavior

have their own specific requirements and methodological pitfalls

to be avoided. So do descriptive models for improving population

segmentation and prescriptive models for recommending best

next actions. In addition, expert (judgment-based) models need

to be documented in very different ways than empirically derived

models. Requirements and methods also vary, of course, across

geographies and markets.

To accommodate this diversity, banks need workflow tools that

incorporate automated business rules management technology.

By authoring and changing business rules, analytics groups can

easily adjust workflows—within standardized parameters and

constraints—to their local needs.

At the same time, centralized repositories enable analytic

teams to share characteristic libraries and learn from each

other’s documentation and validation results. They also support

collaboration, where appropriate, across teams.

FICO® Model Central™ provides consistent end-to-end model

governance and a central repository for all of an organization’s analytic

models. The solution manages models created in any vendor’s tool,

showing their real-time status and monitoring their performance. New

advanced workflow capabilities in version 5.0 orchestrate and manage

model lifecycle processes.

Automated, configurable workflows notify process participants when

they have a new task and provide instant access to the information

needed to complete it. As tasks are performed, workflows capture

key steps, artifacts, decisions and sign-offs. Over time, Model Central

compiles a comprehensive audit trail for model lifecycle processes—

project initiation through data quality testing, model development,

implementation, deployment, validation, and updates or replacement.

Together, these advanced capabilities enable organizations to achieve

peak model performance, while providing the transparency and

accountability required by banking regulators and executives. Without

diverting analytics teams from more valuable work, they help reduce

model risk and raise compliance.

CENTRALIZING MODEL MANAGEMENT

ModelData Mart

Tracking

Monitoring

OngoingValidation

ManagementReporting

Alerts

DecisionSimulation

DecisionExecution

ScoringServices

DecisionOptimization

Development&

Calibration

Deployment&

Verification

ModelData Mart

ADVANCED

P

RO

FESSIONAL

DEC

IS

IONING

DEV

ELOPMENT

FO

UN

DA

TION

Reducing Regulatory Drag on Analytics Teams

INSIGHTS WHITE PAPER

For more information North America Latin America & Caribbean Europe, Middle East & Africa Asia Pacificwww.fico.com +1 888 342 6336 +55 11 5189 8222 +44 (0) 207 940 8718 +65 6422 7700 info@fico.com LAC_info@fico.com emeainfo@fico.com infoasia@fico.com

FICO, Model Central and “Make every decision count” are trademarks or registered trademarks of Fair Isaac Corporation in the United States and in other countries. Other product and company names herein may be trademarks of their respective owners. © 2014 Fair Isaac Corporation. All rights reserved.

4072WP 11/14 PDF

The Insights white paper series provides briefings on research findings and product development directions from FICO. To subscribe, go to www.fico.com/insights.

Tracking model and characteristic lineages

The concept of model lifecycle management is broader than the life of any particular model. As

analytics proliferate across organizations and the pace of change in financial services markets

accelerates, banks need to start thinking in terms of lineages.

Regulators may, for instance, ask pointed questions about why a retired model was replaced

with the current one, and which customer characteristics were given greater predictive weight

in the process. When a customer characteristic is changed by its author/owner, banks need to

know which models incorporate that characteristic so they can manage all downstream effects.

State-of-the-art model lifecycle management takes this broader view. Automated workflows

help banks capture a complete lifecycle history of all models and their components. For each

model, users can quickly track the lineage of any predictive customer characteristic—generated

during development, harvested from a previous model, taken from a shared library, etc. For each

characteristic, they can see everywhere it is currently used or was previously used—predictive

models, segmentation strategies, decision strategies, etc.

Another advantage of this approach is that banks have the opportunity to evaluate the value of

individual customer characteristics over time.

Increasingly far-reaching and detailed regulatory scrutiny is making it more important than ever

for banks to put standard, approved model management processes in place. At the same time,

banking executives want more visibility into and control over the full dimensions of model risk,

including both compliance exposure and performance issues.

Automated workflows that feed model lifecycle management solutions help banks lower model

risk by improving process consistency and accountability. And they do it without clipping the

wings of analytic teams—in fact, they offer efficiencies that can help them soar.

To learn more about best practices for model management, visit the FICO Blog and read these

Insights papers:

• Customer Centricity: Four Bank Success Stories (No. 78)

• Satisfying Customers and Regulators: Five Imperatives (No. 75)

• Comply and Compete—Model Management Best Practices (No. 55)

Conclusion