Post on 22-Oct-2014
Bank of Good Hope
Credit Risk Management
Scorecard Recalibration
Technical Proposal
1
PIC Solutions Presentation Outline
The BGH Business Problem
The BGH Business Goals and Objectives
PIC Solutions: Technical Proposal
Case Study: CRM Scorecard Recalibration
Objectives of the PIC Technical Proposal
Benefits of the PIC Technical Solution
Implementation of the PIC Technical Solution
Discussion
22
The BGH Business Problem Steady Decline of the Credit Card Portfolio
Competition from other Financial Institutions
Outdated Credit Risk Management Scorecards
Deterioration of the CRM Scorecard Performance
Limited Analytical Resources for Model Development
Unacceptably Long Model Development Times
Lack of Confidence in the Decision Analytics Function
Lack of Synergy between Sales and Risk
33
The Required Technical Solution
Updated and Well - Performing CRM Scorecards
Staff Training: SAS and Model Development
Simple Robust and Effective Solution: Easy to Understand Easy to Implement Easy to Maintain
Better Analytical Support for: Credit Risk Management ( Acquisitions to Collections ) Risk Appetite Analysis ( Business Development )
Classification of Customers for Risk-Based Pricing
More Effective and Efficient Credit Risk Management44
PIC Solutions Presentation Outline The BGH Business Problem
The BGH Business Goals and Objectives
PIC Solutions: Technical Proposal
Case Study: CRM Scorecard Recalibration
Objectives of the PIC Technical Proposal
Benefits of the PIC Technical Solution
Implementation of the PIC Technical Solution
Discussion
55
The BGH Business Goals & Objectives
BGH wish to grow the Credit Card Portfolio by using Effective Decision Analytics
They want to implement their Risk Appetite Strategy and Risk-Based Pricing to grow the business and to transform the Credit Card Portfolio.
Existing Staff need to be trained in SAS and Model Development Techniques
The Project Management Office must be established.
BGH are looking for a cost – effective solution and rapid implementation.
They want to see measurable results within 6 months.66
PIC Solutions Presentation Outline The BGH Business Problem
The BGH Business Goals and Objectives
PIC Solutions Technical Proposal
Case Study: CRM Scorecard Recalibration
Objectives of the PIC Technical Proposal
Benefits of the PIC Technical Solution
Implementation of the PIC Technical Solution
Discussion77
The PIC Solutions Technical Proposal
1. Comprehensive Scorecard / Model Evaluation: Data Quality Analysis Model Development Methodology Model Validation Methodology Model / Scorecard Implementation Scorecard Monitoring Reports
2. Credit Card Portfolio Analysis
3. CRM Scorecard Recalibration
4. Updated CRM Scorecard Implementation
5. Risk Appetite Analysis
6. Risk-Based Pricing Strategy
8
The PIC Solutions Technical Proposal
What is Scorecard Recalibration?
Why should we Recalibrate CRM Scorecards? How should we Recalibrate CRM Scorecards?
When should we Recalibrate CRM Scorecards?
How Often should we Recalibrate CRM Scorecards?
Does Scorecard Recalibration New Scorecard Development ?
Will CRM Scorecard Recalibration solve our Analytical Problems?
How Much does Scorecard Recalibration Cost?
99
The PIC Solutions Technical Proposal
Scorecard Recalibration
High Level Overview: The Objectives of Scorecard Recalibration
The Benefits of Scorecard Recalibration
The Scorecard Recalibration Process
Detailed Description: The Scorecard Recalibration Methodology
Analytical Support for Scorecard Recalibration
The Scorecard Recalibration Toolkit
1010
Scorecard Recalibration Objectives
Identify those Scorecards where there has been significant
deterioration in the performance of the Scorecard
Explore the potential value in recalibrating the existing Scorecards
using the most recently available population data without changing
the Structure of the Scorecard and its Component Variables.
If necessary, Recalibrate the Scorecard Model as required in an
incremental and phased approach.
Facilitate the rapid development of CRM Strategy based on the
Recalibrated Scorecard
1111
Scorecard Recalibration Objectives
Overcome the Analytical Resource Constraint.
Evaluation of the CRM Scorecard on Current Data
Updating the Model Parameters
Incremental Modification of the Model as required
Improvement of the Scorecard / Model Performance
Reduction in NEW Model Development Costs
Rapid Scorecard Development
Rapid Scorecard Implementation
Robust, Effective and Simple Solution1212
Scorecard Recalibration Benefits
Model Risk Reduction
Credit Risk Management and Business Support
Reduction in Model Development Costs
Reduction in Model Development Project Time
Resource Development
Tactical Flexibility
Incremental Change Management
Strategy Development
Facilitates Portfolio Transformation1313
So what is the Real Problem ? Most Credit Risk Scorecards are based on a Statistical Model.
The Statistical Model is based on several assumptions.
The Statistical Model extracts information from the Data.
The quality of the data will affect the power of the model.
The quality of the model development process is critical.
The performance of the model is measured by several statistics.
The application of the scorecard affects the statistical measures.
The business and risk assumptions may no longer be valid.
The intended target population is always changing.
The Scorecard is Out of Date BEFORE it is Implemented.
1414
CRM Scorecard Models
Most Credit Risk Scorecards are based on a Statistical Model:
The Model uses the information contained within a linear combination of the Predictor Variables.
The individual Predictor Variables in the Model have a Linear or Monotonic relationship with the Binary Target Variable
The Model is fitted to the Portfolio Population using Binary Logistic Regression
The Model estimates the probability that the Binary Outcome Variable will take on one of its two values ( Default or Not-Default)
The Model Development process produces a set of weights
corresponding to the attributes within each characteristic variable which is included in the Model.
The values / range of the Attribute Weights reflect the Predictive and / or Classification Power of the respective Characteristics.
1515
CRM Scorecard Model Assumptions
The fundamental assumptions behind all predictive credit risk scoring systems are:
The past behaviour of one group of customers can be used to
predict the future behaviour of another group of customers with
similar credit risk profiles (characteristics ).
The future macro-economic and political environment ( in which
the Model will be applied ) will be generally the same as the past
economic environment in which the Model was developed.
The rate of change within the bank’s credit risk portfolio over a
given time period is gradual, smooth and continuous.
Small changes in Input produce consistent changes in Output. 1616
CRM Scorecard Model Assumptions
Unfortunately, these assumptions are often NOT TRUE;
The world is always changing and such change is seldom gradual,
smooth and continuous.
Forecasting the future is a difficult task, especially when shocks to
the system disrupt trends and increase volatility.
Many Events or Factors that influence Customer Default Behaviour
cannot be predicted from the Customer’s Credit Risk Profile.
Given that Credit Risk Scorecards are built using historical data in
order to help predict future behaviour, all scoring systems will lose
Predictive Power over time and eventually require redevelopment.
To maintain and enhance the performance of a scoring system, its
components must be validated on an ongoing basis so that any
deterioration can be identified and corrected.
1717
CRM Scorecard Performance Measures
The Performance of a Credit Risk Scorecard is best evaluated using measures that are specifically related to its application within the Credit Risk Management System:
Rank-Ordering Classification Prediction Explanation
The standard diagnostic tools can be used to measure the Reliability and Predictive Power of a Credit Risk Scoring system across the following three dimensions:
Scorecard and Characteristic Discriminatory Power Scorecard and Characteristic Stability Cut-off Score Performance ( requires Cost Information )
1818
CRM Scorecard Performance Measures
Scorecard and Characteristic Discriminatory Power:
The Model Validation system measures the Power of the scoring system ( and each of the characteristics in the Scorecard ) to separate the "good risk" and "bad risk" sub-populations.
Various statistical performance measures, such as the Gini and
the Kolmogorov-Smirnov (K-S) test, are used to measure the ability of the score to separate “goods” and “bads” and provide the expected "lift" between low and high scoring segments.
The Gini is best used to compare the Power of two or more Binary Decision Scorecard Models derived from the same portfolio population where the costs of the two decision errors are the same for each model.
1919
CRM Scorecard Performance Measures
Scorecard and Characteristic Discriminatory Power:
However, the Gini is NOT a coherent measure of the Power of a Binary Decision Scorecard Model unless the costs of each decision option are
included in the Model Evaluation process.
Type 1 Error: Decision: Reject a Good Customer Cost is related to the loss of potential revenue gain,
Type 2 Error: Decision: Accept a Bad Customer Cost is related to the loss due to possible bad debts.
This is even more important when the Gini measure is used to compare two or more Scorecards each from a different portfolio / population where the decision costs are quite different.
20
Portfolio Type 1 Error Cost Type 2 Error Cost
Hire Purchase
Mortgages
20
CRM Scorecard Monitoring
Scorecard and Characteristic Stability: Economic climates and business environments are constantly changing and this
can influence both the structure and the composition of a credit risk portfolio. These changes will impact the population distributions (with respect to key credit
risk drivers) as well as the interactions between the factors and consequently the changes affect the performance of a credit risk scoring system.
To determine if a scoring system can continue to be used effectively, it is
necessary to measure the stability of the target population relative to both: the Model Development Population the Current Scorecard.
The Population Stability Index measures the degree of change within the target
population, both for the overall scoring system and for each characteristic in the Scorecard.
High Population Stability Index values indicate the population has changed and may suggest that the entire scoring system or a specific component needs Recalibration or redevelopment.
2121
CRM Scorecard Challenges
Given the small size of some Retail Credit portfolios as well as the resulting
low number of “Events” vs. “Non-Events” in the Binary Outcome Variable
that is being Modelled;
The Model Development Method has to overcome several challenges:
Business Requirements for Portfolio Segmentation.
Strong Predictor Variables which may not be “Good for Business”.
Sparse data sets with respect to Predictor Variables and / or the Target Outcome Variable.
“Clumpy” distributions of the “Event” across one or more Predictor Variables.
Uncertain Data Quality during periods of economic flux and portfolio volatility.
Insufficient (historical) portfolio data for robust Scorecard Development.
Scorecard Validation Data requirements may aggravate the above problems.
2222
OK; So what is the Solution?
Accept the limitations of Credit Risk Scorecards.
Evaluate the Statistical Model Assumptions.
Engage the Business in Credit Risk Model Development.
Do Exploratory Data Analysis and Data Preparation for MD
Use a well structured Model Development Methodology
Use a well structured Model Validation Methodology
Select appropriate model performance measures.
Evaluate the application of the scorecard using monitoring.
Frequently Update and Recalibrate each CRM Scorecard.
Evaluate the factors which impact on the CRM Scorecard.
2323
CRM Scorecard Recalibration Methodology
If the Scorecard does not adequately validate on the most recent Portfolio Data, then we will provide recommendations to address any deficiencies that are identified during the analysis.
We are able to assist you with the necessary remedial action, which can include one or more of the following solutions:
Simple re-weighting of the existing Scorecard characteristics and their
attributes.
Optimisation of the attribute structure of the Scorecard characteristics.
Identification and replacement of one or more problematic variables.
Low Default Portfolio Analysis and Model Recalibration
Complete New Scorecard Development.
2424
CRM Scorecard Recalibration Methodology
The Specific Scorecard Recalibration Methodology that is applied will depend on:
The type of Scorecard that is being investigated:
– Application Scorecards ( May impact BASEL II PD Models )– Reject Inference, – Categorical Variables – Integer Scores
– Behaviour Scorecards ( May impact BASEL II PD Models )
– Collections Scorecards
– Recoveries Scorecards
The Structure of the Business Portfolio
The Application of the Scorecard within Risk Management
The Availability of Credit Bureau Data
2525
CRM Scorecard Recalibration Process
Phase 0: ( New Validation )Evaluate the Existing Scorecard on Recent Data
Phase 1: ( New Weights )Keep the Existing Scorecard Structure Intact
Phase 2: ( New Attributes ) Optimise The Attribute Structure of The Variables
Phase 3: ( New Variables ) Replace Weak or Problematic Scorecard Variables
Phase 4: ( New Target Definition ) Low Default Portfolios, RELAX the BAD Definition
Phase 5: ( New Model ) Develop a completely NEW Scorecard Model
2626
CRM Scorecard Recalibration: Phase 0
Phase 0: (Evaluate the Existing Scorecard )
Retain the same underlying Statistical Model.
Retain the same Binary Target Variable Definition
Retain the same Scorecard Characteristic Variables
Retain the same Attribute Structures within each Characteristic
Retain the same Scorecard Weights for each Attribute.
Construct a suitable Data Set for OOT Model Validation.
Evaluate the Existing Scorecard / Model on Recent Data2727
CRM Scorecard Recalibration: Phase 1
Existing Scorecard Recalibration:Compute New Scorecard Weights
Retain the same underlying Statistical Model
Retain the same Binary Target Variable Definition
Use the previously constructed Data Set for OOT Model Validation
Keep the Existing Scorecard Structure Intact – Retain the same Scorecard Characteristic Variables– Retain the same Attribute Structures within each
Characteristic Variable
Compute New Scorecard Weights ( Points ) for each Attribute (and therefore for each Characteristic Variable)
2828
CRM Scorecard Recalibration: Phase 2
Optimise the Attribute Structure for each Characteristic Variable
Retain the same underlying Statistical Model
Retain the same Binary Target Variable Definition
Keep the Existing Scorecard Structure Intact ( Characteristics )
Retain the same Scorecard Characteristic Variables
Optimise the Attribute Structures within each Characteristic Variable
Compute New Scorecard Weights ( Points ) for each Attribute (and therefore for each corresponding Characteristic Variable)
2929
CRM Scorecard Recalibration: Phase 3
Replace Problematic Variables
Retain the same underlying Statistical Model
Retain the same Binary Target Variable Definition
Modify the Existing Scorecard Structure ( Characteristics and / or Attributes )
Replace the Weak or Problematic Characteristic Variables
Optimise the Attribute Structures within each Characteristic Variable
Compute New Scorecard Weights ( Points ) for each Attribute (and therefore for each corresponding Characteristic Variable)
3030
CRM Scorecard Recalibration: Phase 4
LDP Construct a New Target Variable Definition
Relax the Binary Target Variable Definition to create more positive cases (Defaults) and then apply Phases 1 , 2 & 3.
Modify the Existing Scorecard Structure ( Characteristics and / or Attributes )
Replace the weak or problematic Characteristic Variables
Optimise the Attribute Structures within each Characteristic
Compute New Scorecard Weights ( Points ) for each Attribute (and therefore for each corresponding Characteristic Variable)
3131
CRM Scorecard Recalibration: Phase 5
Build a NEW Scorecard Model
Follow BGH Group Standards for CRM Models
Construct the Binary Target Variable Definition
Construct the Characteristic Variables and Attributes
Optimise the Attribute Structure within each Variable
Select the best Characteristic Variables for the Model
Fit the Binary Logistic Regression Model
Compute the Scorecard Weights for each Attribute and Characteristic Variable included in the model
Evaluate and Validate the Scorecard Model
3232
PIC Solutions Presentation Outline The BGH Business Problem
The BGH Business Goals and Objectives
PIC Solutions Technical Proposal
Case Study: CRM Scorecard Recalibration
Objectives of the PIC Technical Proposal
Benefits of the PIC Technical Solution
Implementation of the PIC Technical Solution
Discussion 3333
Case Study: HP Application Scorecard
The Hire Purchase Application Scorecard was developed on 2005-2007 data and implemented in Nov 2009
Recent monitoring of the Scorecard has shown a significant
deterioration in the scorecard in terms of: Discriminatory Power (Gini coefficient) Significance of Scorecard Characteristics Risk Ranking by Score
The Existing Scorecard includes a variable which is having a detrimental effect on the size of the portfolio
Business Requires an Urgent Fix while waiting for the complete Redevelopment of the Existing Suite of Scorecards
3434
Existing HP Application Scorecard
Gini: 11.9Mar 09
35
Variable Attribute BandScore
Points
Total Monthly Income <=418 115
>418 170
Salary & BGH Flag Non BGH Account Holder 121
BGH holder - non salary transfer 160
BGH holder & salary transfer 195
Age < 26 or missing 78
26 - 48 or 53+ 96
48 - 53 126
# of Bank Accounts < 1 108
>= 1 149
Nationality South African 58
Other 86
Photo Card Other 98
Photo card 140
Monthly Expenses > 0 156
<= 0 200
35
ROC Curve for Model Evaluation
0
20
40
60
80
100
0 20 40 60 80 100% Goods
% B
ads
0
20
40
60
80
100
0 20 40 60 80 100% Goods
% B
ads
ROC Curve for Model Evaluation
ROC Curve for Model Evaluation
0
20
40
60
80
100
0 20 40 60 80 100% Goods
% B
ads
0
20
40
60
80
100
0 20 40 60 80 100% Goods
% B
ads
ROC Curve for Model Evaluation
0
20
40
60
80
100
0 20 40 60 80 100% Goods
% B
ads
ROC Curves for Comparison of 2 Scorecards
Existing HP Scorecard Evaluation
Gini: 11.90
Performance of the Existing Scorecard on the Present 2011 Portfolio
• Scorecard no longer performing on the current population
• The BGH flag variable has a negative effect on the portfolio composition
• The Photo Card Variable is not used and data is no longer captured
4141
Recalibration Phase 01- Results
Gini: 44.65
42
Variable Attribute Score Points
Intercept 392Total Monthly Income <=418 -22
>418 23Age <26 or missing 3
26-48 or 53+ 048-53 -2
Monthly Expense <=0 0>0 0
Salary & BGH Flag Non BGH Account Holder -32BGH Holder- non salary transfer -12BGH Holder- salary transfer 163
Nationality South African -36Other 13
Photo Card Flag No Photo Card 3Photo Card -48
Number of Bank Accounts >=1 1<1 -23
42
Recalibration Phase 2 - Results
Gini: 40.38 Gini: 44.65
43
Variable Attribute Score Points
Intercept 392Total Monthly Income <=245 -48
245-299 -22300-449 14
450+ 19<=26 -1027-29 -1429+ 5<=0 40-79 -3180-99 -16100+ 14
Salary Transfer & BGH Customer Non BGH Account Holder -32BGH Holder- non salary transfer -12BGH Holder- salary transfer 162
Nationality Malaysian -31Other 11
Photo Card Flag No Photo Card 3Photo Card -47
Number of Accounts >=1 1<1 -23
Age
Monthly Expense
43
Recalibration Phase 3 - Results
Gini: 49.6
44
Variable Attribute Score Points
Intercept 391
Total Monthly Income <=244 -56
245-299 -26
300-449 17
450+ 23
Monthly Expenses <=0 5
0-79 -43
80-99 -23
100+ 20
Number of Accounts >=1 -25
<1 11
Salary Mode Salary Transfer -17
Non Salary Transfer 34
Nationality South African -32
Non-SA African -55
Other 16
N -25
Y 51
Worktype Salary 5
Self -Employed & Other -124
Standing Instruction Set-Up Flag
44
Final Recalibrated Scorecard - Results
All the attributes within each scorecard characteristic now have significant points
Age variable no longer significant
Photo Card removed
Maximum separation between attribute points to give optimum distribution of scores
BGH & Salary transfer has been replaced by Salary Mode, and worktype
Standing instruction set up has also become significantGini: 49.6
4545
Recalibration Phase 1 - 3: Results
Gini: 40.38 Gini: 44.65
Model Phase Gini KS
Current 0 11.90
Re-Aligned 1 40.38
AttributesOptimised
2 44.65
New Variables
3 49.60
4646
Final Recalibrated Scorecard
ROC Curve
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.2 0.4 0.6 0.8 1
Cum % Goods
Cu
m %
Ba
ds
New
Random
Original
KS
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
170 220 270 320 370 420 470 520
Score
Cu
m %
Acc
ou
nts
Cum % Bads
Cum % Goods
Score Band
Total Accounts
% Accounts Cum % Goods BadsCum % Bads
Bad Rate
low-327 545 11% 11% 462 83 37.22% 15.23%328-370 551 11% 21% 508 43 56.50% 7.80%371-376 458 9% 30% 443 15 63.23% 3.28%377-418 687 13% 44% 665 22 73.09% 3.20%419-424 663 13% 57% 634 29 86.10% 4.37%425-437 199 4% 61% 194 5 88.34% 2.51%438-451 489 10% 70% 476 13 94.17% 2.66%452-482 521 10% 80% 516 5 96.41% 0.96%483-500 689 13% 94% 681 8 100.00% 1.16%501-high 324 6% 100% 324 0 100.00% 0.00%Total 5126 100% 4903 223 4.35%
Scorecard ranks bad accounts• 1st score band has bad rate 15 %
• 37% of bad accounts captured in 1st score band
• 56% of bad accounts captured in first two score bands
4747
PIC Solutions Presentation Outline The BGH Business Problem
The BGH Business Goals and Objectives
PIC Solutions Technical Proposal
Case Study: CRM Scorecard Recalibration
Objectives of the PIC Technical Proposal
Benefits of the PIC Technical Solution
Implementation of the PIC Technical Solution
Discussion
4848
PIC Solutions Presentation Outline The BGH Business Problem
The BGH Business Goals and Objectives
PIC Solutions Technical Proposal
Case Study: CRM Scorecard Recalibration
Objectives of the PIC Technical Proposal
Benefits of the PIC Technical Solution
Implementation of the PIC Technical Solution
Discussion
4949
PIC Solutions Presentation Outline The BGH Business Problem
The BGH Business Goals and Objectives
PIC Solutions Technical Proposal
Case Study: CRM Scorecard Recalibration
PIC Solution Objectives
Benefits of our Technical Solution
Implementation of the Technical Solution
Discussion5050
Implementation of the Technical Solution
CRM Scorecard Evaluation ( Phase 0 )
CRM Scorecard Recalibration ( Phases 1 to 5 )
CRM Scorecard Validation ( Phase 6 )
Implementation of the Recalibrated Scorecards
Monitoring of the New Scorecard Performance
Implementation of the CRM Strategy
Monitoring of Portfolio Transformation
5151
CRM Scorecard Implementation
Cut-off Score Performance:
A scoring system can continue to rank order the population and provide acceptable discriminatory Power, yet fail to produce the desired credit risk management performance.
Scorecard Cut-Offs are specific score values that are selected when
designing decision rules to automate credit risk decisions that are based on the Scorecard.
Shifts in Scorecard performance by score range may demand adjustments to the score cut-offs and the associated decisions.
Scorecard validations include updated score performance tables for each key portfolio segment so that score cut-offs can be adjusted in line with both credit risk and business strategy.
Cut-Off Analysis should also include the Utility-Costs of each decision
that is based on whether or not the score is above the cut-off value.
5252
Cut-Off Analysis – New Scorecard
We have decreased the overall Bad Rate within the Approved Groups from 4.4% to 3.5%, while the Approval Rate has been decreased from5126 to 5012
Current Cutoff- 293
Approve Decline SummaryGoods 4,480 423 4,903 Bads 152 71 223 Total 4,632 494 5,126
Bad Rate 3.3% 14.4% 4.4%Goods 359 150 509 Bads 21 23 44 Total 380 173 553
Bad Rate*** 5.5% 13.2% 7.9%Goods 4,839 573 5,412 Bads 173 94 267 Total 5,012 667 5,679
Bad Rate 3.5% 14.1% 4.7%
Summary
New Scorecard
Existing Scorecard
Approve
Decline**
5353
Cut-Off Analysis – New Scorecard
Keeping the approval rate at 90%, we can reduce the bad rate to 3.6%
Insert $ value of scorecard
Current Approval Rate- 90%(Score-291)
Approve Decline SummaryGoods 4,533 370 4,903 Bads 155 68 223 Total 4,688 438 5,126
Bad Rate 3.3% 15.5% 4.4%Goods 456 49 505 Bads 33 15 48 Total 489 64 553
Bad Rate*** 6.8% 23.6% 8.7%Goods 4,989 419 5,408 Bads 188 83 271 Total 5,177 502 5,679
Bad Rate 3.6% 16.5% 4.8%
New Scorecard
Existing Scorecard
Approve
Decline**
Summary
5454
Cut-Off Analysis – New Scorecard
Keeping the Bad Rate at 4.4%, we can increase the Accept Rate to 97%
Current Bad Rate- 4.4% (Score-192)
Approve Decline SummaryGoods 4,835 68 4,903 Bads 206 17 223 Total 5,041 85 5,126
Bad Rate 4.1% 20.0% 4.4%Goods 479 31 509 Bads 35 8 44 Total 514 39 553
Bad Rate*** 6.9% 21.1% 7.9%Goods 5,314 99 5,412 Bads 241 25 267 Total 5,555 124 5,679
Bad Rate 4.3% 20.4% 4.7%
New Scorecard
Existing Scorecard
Approve
Decline**
Summary
5555
Strategy Results (Credit Limit)Difference Report – after 12 months
30.72%
21.54%
25.47%
7.06%
26.59%
4.05%
-1.51%
-29.49%
CHALLENGERCHAMPION CHANGE
Total Current Balance (DR)
Average Current Balance (DR)
Cash Sales
Merchandise Sales
Finance Charges
Increase in % Current Balances
Reduction in % 2Cyc Balances
Over-limit Accounts
CHALLENGERCHAMPION CHANGE
Total 2 Cycle Balance
Total 3 Cycle Balance
Total 4+ Cycle Balance
Roll Rates 2 – 3 Cycle (Accts)
Roll Rates 2 – 3 Cycle (ZAR)
Roll Rates 3 – 4+Cycle (Accts)
Roll Rates 3 – 4+ Cycle (ZAR)
Total Current Balance
Strategy Results (Delinquency)Difference Report – after 12 months
-10.69%
-30.77%
-33.78%
-27.42%
-31.00%
-39.74%
-29.12%
30.72%
Conclusion
How will PIC deliver the required value to BGH?
Practical Vehicles for Delivery: Programme Management
Multiple Projects Project Integration
Project Based Management Terms of Reference Project Design Project Scope Project Plan Project Resources
58
Conclusion
What direct benefits can the PIC Methodology Provide to the Bank of Good Hope?
More Effective and More Efficient Credit Risk Management Account Acquisition Account Management Account Collections
Reduction in Credit Risk Loss Increase in Customer Profitability Practical Training and Skills Development
59
Conclusion
How will PIC deliver the required value to BGH?
General Principles:
Consultancy ( We will listen and understand)
Engagement ( We will empower you )
Collaboration ( We will work together )
Knowledge Transfer ( We will share our expertise )
Partnership ( All of the above )
60
Conclusion 1. Doing the Right Thing
Data Driven Quantitative Score Foundation CRM Policy Business Strategy
2. For the Right Reason Risk Appetite Framework Business Objectives
3. At the Right Time Quantitative Analysis Economic Factors Business Knowledge
6161
Questions
and
Discussion
62
Questions and Discussion
The Effective Use of CRM Scorecards
CRM Scorecard Model Development
CRM Scorecard Model Validation
The PIC Technical Proposal: Recalibration
Project Management and Delivery
Development and Implementation in SAS
Staff Training and Development
63
CRM Scorecard Recalibration
The Scorecard Recalibration Methodology
The Objectives of Scorecard Recalibration
The Benefits of Scorecard Recalibration
The Scorecard Recalibration Process: DETAILS
Analytical Support for Scorecard Recalibration
The Scorecard Recalibration Toolkit
6464
Scorecard Recalibration Phase 0
Evaluate the Existing Scorecard on Recent Data
Data Management:
Request / extract the most recent data set that is required for the specific scorecard evaluation and Recalibration analysis.
If necessary, import the RAW Data into SAS to construct a SAS Data Set.
Obtain and / or construct the Metadata Dictionary from the Business Intelligence Unit.
Conduct Exploratory Data Analysis and Data Cleaning ( Data Quality ).
Verify or Construct the Good-Bad Binary Dependent Variable ( Model Development Definition ).
If necessary, construct the Scorecard Characteristic Variables and the corresponding component attributes ( categories ) in the SAS Data Set.
65
Scorecard Recalibration: Phase 0
Evaluate the Existing Scorecard on Recent Data
1. Score each account in the portfolio using the existing Scorecard;
Translate the Existing Scorecard (Characteristics, Attributes, Weights and Intercept) into SAS Program Code.
Compute the Score for each account in the most recent Population Data Set.
2. Scorecard Model Evaluation:
Use the GINI MACRO and the Scores from the Current Data Set to compute the Gini in order to Measure the Power of the Scorecard to separate “Goods” and “Bads”.
3. Model Performance Measures: [ Classification and Prediction ]
Gini > BGH Group Standard ( Specific Portfolio + Model Type ) ?
66
Scorecard Recalibration: Phase 0
Evaluate the Existing Scorecard on Recent Data Request / extract the most recent data set required for the specific analysis.
Population Stability Analysis:
Compute the Population Stability Index ( Relative to the Scorecard Model ) to
determine whether the Current Portfolio Population has drifted away from the
Model Development Population that was used to construct the scorecard?
Scorecard Characteristic Stability Analysis:
Compute the Characteristic Stability Index, ( Relative to each Individual
Characteristics in the Scorecard ) to determine whether the Current Portfolio
Population has drifted away from the Model Development Population that
was used to construct the scorecard?
Construct the Existing Scorecard Baseline Performance Report..
6767
Scorecard Recalibration: Phase 0
Actions after completing Phase 0
Successful Evaluation: Recalibration Decision: STOP
If the Scorecard Performance as measured in the above steps is GOOD or adequate
by BGH Standards and there is no evidence of any fundamental problems with the
Scorecard then STOP and do not proceed to Recalibration Phase 1.
Unsuccessful Evaluation: Recalibration Decision: GO
If the Scorecard Performance as measured in the above steps is NOT GOOD or
adequate by BGH Standards OR there is evidence of fundamental problems with
the Scorecard then proceed to Recalibration Phase 1.
6868
Scorecard Recalibration Phase 01a
Keep the Existing Scorecard Structure Intact
Data Management: ( in addition to Phase 0 )
Based on the Outcome of the Characteristic Stability Analysis and the Scorecard Model Evaluation Phase 0;
Conduct Exploratory Data Analysis and Data Cleaning
Verify or Reconstruct the Good-Bad Binary Dependent Variable
Verify or reconstruct the Scorecard Characteristic Variables and their component attributes.
69
Scorecard Recalibration Phase 01b
Keep the Existing Scorecard Structure Intact
For each Nominal or Categorical Characteristic Variable;
Allocate integer values to each distinct category in the Characteristic Variable.
Compute the cross tabulation frequencies of each category with the Good-Bad variable.
Use these frequencies to compute the Weights of Evidence for each attribute.
Map the Weights of Evidence onto the (linear integer scale) created above.
Scorecard Model Evaluation:
Fit a Logistic Regression Model to the WOE-Transformed Scorecard Variables.
Evaluate the Full Logistic Regression Model ( All Variables )
70
Scorecard Recalibration Phase 01c
Keep the Existing Scorecard Structure Intact
Model Performance Measures:
Evaluate the Scorecard Variables by comparing the AIC and SBC values for the Intercept Only Model with the corresponding AIC and SBC values of the Full Model.
Classification Model Chi-Square: p-value < 0.05 is GoodGini > BGH Group Standard ( Specific Portfolio + Model Type ) ?KS > BGH Group Standard ( Specific Portfolio + Model Type ) ?
Prediction H-L Goodness of Fit: p-value > 0.05 is Good Parameter Estimates should be large and p values should be small Compare the p-values together with the Information Values
– P-value small and large IV Strong– P-value large and small IV Weak
71
Scorecard Recalibration Phase 1
Keep the Existing Scorecard Structure Intact
Actions Required for Case A: Successful Recalibration
If the performance of the recalibrated Model is significantly better than that of the original development Model and there has been a significant population shift since development, ( PSI is relatively large) then:
Confirm that each individual characteristic variable in the Model is statistically
significant and Predictive.
Recalibrate the existing Scorecard using the Logistic Regression Model
parameter values obtained from the most recent population data set.
Align the recalibrated Scorecard to the BGH standard
Do Cut-off analysis to select the appropriate cut-off score value in order to
construct the relevant credit risk decision rule.
72
Scorecard Recalibration Phase 1
Keep the Existing Scorecard Structure Intact
Case B: Unsuccessful Recalibration in P_01 [ Go To Phase 2 ]
If the performance of the Recalibrated Model is NOT significantly better than that of the existing Development Model, then:
Investigate the Predictive Power of the individual Characteristic Variables included in the existing Scorecard (relative to the current population data.)
Investigate the Attribute Structure of each Characteristic Variable that is included in the existing Scorecard (relative to the current population data.)
Identify Weak Predictor Variables that could be excluded from the Scorecard Model.
Identify Problematic Predictor Variables that could be excluded from the Scorecard Model: Bad for Business Data No Longer Collected or Available Product Change Data Quality (Missing Data, Outliers, Errors, Collinearity, Insufficient Spread)
73
Scorecard Recalibration Phase 1d
Go to Recalibration Phase 2.
If the Recalibrated Model from Phase 1 is significantly
better than the existing model but there are one or
more weak or problematic characteristic variables in
the Phase 1 Recalibrated Model.
If the performance of the Recalibrated Model from
Phase 1 is not adequate or there are structural
problems with the Phase 1 Recalibrated Model.
74
Scorecard Recalibration Phase 02a
Optimise The Attribute Structure of The Variables
Retain the Existing Scorecard Characteristic Variables. For each Continuous Scorecard Characteristic Variable:
Optimise the Attribute Structure ( Bins or Categories ) in order to maximise the univariate Predictive Power of the Characteristic Variable in relation to the Binary Default Variable. (measured by the Gini)
Classify the observed values of the Characteristic Variable into the appropriate ordered set of attribute categories ( Bins ).
Map an ordered set of integer values to the corresponding ordered set of distinct attribute categories ( Bins ) within the Characteristic Variable.
For each Categorical Scorecard Characteristic Variable: Investigate the Attribute Structure of the Categorical Characteristic Variables. Where it is feasible to modify the attribute structure of a categorical
characteristic variable, apply the same optimisation process as described above.
75
Scorecard Recalibration Phase 02b
Optimise The Attribute Structure of The Variables
Retain the Existing Scorecard Characteristic Variables.
For all Scorecard Characteristic Variables:
Compute the frequencies of each New Attribute Category in a cross tabulation with the Binary Default Variable.
Use these frequencies to compute the Weights of Evidence for each attribute.
Map the Weights of Evidence onto the ( linear integer scale ) created in steps 2 or 3
Fit a Binary Logistic Regression Model to the WOE-Transformed Scorecard Characteristic Variables:
Force all the existing Variables into the Full Model Perform Stepwise Variable Selection to compare the Full Model
with the Subset Model
76
Scorecard Recalibration Phase 02c
Optimise The Attribute Structure of The Variables
(Retain the Existing Scorecard Characteristic Variables.) Evaluate the Logistic Regression Model using the SAS Output:
AIC and SBC should decrease with the addition of useful explanatory Variables into the Regression Model during stepwise selection.
Model Chi-Square: p-value < 0.05 is Good
Gini and KS should exceed BGH Group Standards
H-L Goodness of Fit: p-value > 0.10 is Good
Parameter Estimates should be large and p values should be small.
Compare the p-values with the corresponding Information Values
77
Scorecard Recalibration Phase 02d
Optimise The Attribute Structure of The Variables
(Retain the Existing Scorecard Characteristic Variables.)
Case A: Successful Recalibration
If the performance of the recalibrated Model ( based on the new attribute structures ) is significantly better than that of the development Model, then do the following:
Confirm that each individual characteristic variable in the Model is statistically significant and Predictive.
If so then retain the existing Scorecard Characteristic Variables. Apply the new Attribute Structures within each Scorecard Characteristic Variable. Recalibrate the existing Scorecard using the parameter values obtained from the
Logistic Regression Model that has been fitted to the most recent data set. Align the recalibrated Scorecard to the BGH standard Conduct Cut-off analysis to select the appropriate cut-off score value(s) for the
credit risk decision rule(s).
78
Scorecard Recalibration Phase 02e
Optimise The Attribute Structure of The Variables
(Retain the Existing Scorecard Characteristic Variables.)
Case B: Unsuccessful Recalibration in P_02 [ Go To Phase 3 ]
If the performance of the Recalibrated Model ( using the modified attribute structures ) is NOT significantly better than that of the Development Model, then:
Existing Scorecard Investigate the Univariate Predictive Power of the individual Characteristic
Variables included in the existing Scorecard (current population data). Investigate the Multivariate Predictive Power of various groups of Characteristic
Variables included in the existing Scorecard (current population data).
Modified Scorecard Investigate the Univariate Predictive Power of the individual Characteristic
Variables NOT included in the existing Scorecard (current population data). Investigate the Multivariate Predictive Power of various groups of Characteristic
Variables NOT included in the existing Scorecard (current population data).
79
Scorecard Recalibration Phase 02f
Go to Recalibration Phase 3.
If the Recalibrated Model from Phase 2 includes
one or more weak or problematic characteristic
variables which should be replaced.
If the performance of the Recalibrated Model from
Phase 2 is not adequate.
There are structural problems with the Phase 2
Recalibrated Model.
80
Scorecard Recalibration Phase 03a
Replace Weak or Problematic Scorecard Variables
If the performance of the Phase 2 Recalibrated Model is NOT significantly better than that of the Development Model, then: Investigate the Predictive Power of the individual Characteristic Variables included in
the existing Scorecard (relative to the current population data.)
Investigate the Attribute Structure of each Characteristic Variable that is included in the existing Scorecard (relative to the current population data.)
Identify Weak Predictor Variables that could be excluded from the Scorecard Model.
Identify Problematic Predictor Variables that could be excluded from the Scorecard Model: Bad for Business Data No Longer available Product Change Data Quality (Missing Data, Outliers, Errors, Collinearity, Insufficient Spread)
81
Scorecard Recalibration Phase 03b
Replace Weak or Problematic Scorecard Variables
Identify those Characteristic Variables in the Recalibrated Model from Phase 2 that are weak Predictors and do not contribute to the Predictive Power of the Scorecard Model:
Univariate Analysis ( Xk: k = 1 to n ) WOE Information Value Variance
Bivariate Analysis Time Series ( Xk , Time ) k = 1 to n Logistic Regression ( Y , Xk ) k = 1 to n
Multivariate Analysis Logistic Regression ( Y , X1 ,X2 ,X3 ,...., Xn) Gini and KS AIC and SBC Interactions / Correlation / Collinearity
82
Scorecard Recalibration Phase 03c
Replace Weak or Problematic Scorecard Variables
Investigate alternative Explanatory Variables that are NOT in the
Recalibrated Model from Phase 2 but which could be used to replace
the weak or problematic Characteristic Variables in the Scorecard.
Use the same analytical techniques as described in previous slides in
order to ensure that the NEW Variables are compared with the OLD
Variables against the same measures so that the potential benefits of
swapping the two sets of Variables is evident.
Replace the weak Predictor Variables presently in the Recalibrated Model
from Phase 2 with strong(er) Predictor Variables that were identified in the
above Phase 3 analysis.
83
Scorecard Recalibration Phase 03d
Replace Weak or Problematic Scorecard Variables
The ranking and selection of NEW Variables for inclusion in the Phase 3
Recalibrated Scorecard should be based on the following factors:
Variables that are well behaved Statistically Variables that satisfy the Logistic Regression Model Assumptions Variables that have Univariate Predictor Power Variables that have Multivariate Predictor Power Variables from different Variable Clusters ( Risk Factors ) Variables that make good Business Sense ( Behaviour and Portfolio
Transformation)
If necessary, supplement the Recalibrated Scorecard with a set of Expert
Decision Rules that are based on key risk drivers which are not explicitly
included as Explanatory Variables in the Model.
84
Scorecard Recalibration Phase 03e
Replace Weak or Problematic Scorecard Variables Construct WOE-Transformed Scorecard Variables for Model Development:
Fit a Logistic Regression Model to the WOE-Transformed Scorecard Variables: Force all the Variables from the Existing Scorecard into the Full Model Perform Stepwise Variable Selection to compare the Full Model with the
Subset Model(s) Evaluate the NEW Logistic Regression Model using the SAS Output:
AIC and SBC should decrease with the addition of useful Explanatory Variables into the regression model during Stepwise Variable Selection.
Model Chi-Square: p-value < 0.05 is Good Gini and KS should exceed BGH Group Standards H-L Goodness of Fit: p-value > 0.10 is Good The Model Parameter Estimates should be large and p values should be small. Compare the Parameter p-values with the corresponding Information Values
85
Scorecard Recalibration Phase 03f
Replace Weak or Problematic Scorecard Variables
Case A: Successful Recalibration : STOP
If the performance of the Recalibrated Model from Phase 3 is significantly better than that of the Phase 2 Recalibrated Model from, then do the following:
Confirm that each individual Characteristic Variable in the Model is Statistically significant and Predictive.
If so then retain the existing Phase 3 Scorecard Characteristic Variables.
Apply the new Attribute Structures within each Characteristic Variable.
Recalibrate the existing Phase 3 Scorecard using the parameter values obtained from the logistic regression Model that has been fitted to the most recent data set.
Align the Phase 3 Recalibrated Scorecard to the BGH standards
Conduct Cut-off analysis to select the appropriate cut-off score for the credit risk decision rule.
86
Scorecard Recalibration Phase 03g
Replace Weak or Problematic Scorecard Variables
Case B: Unsuccessful Recalibration If the Recalibrated Model from Phase 3 is NOT significantly better than the Phase 2 Recalibrated Model then do the following: Existing Phase 3 Scorecard Variables
Investigate the Univariate Predictive Power of the individual Characteristic Variables included in the existing Scorecard (current population data).
Investigate the Multivariate Predictive Power of various groups of Characteristic Variables included in the existing Scorecard (current population data).
Potential Phase 3 Scorecard Variables Investigate the Univariate Predictive Power of the individual Characteristic
Variables NOT included in the Phase 3 Scorecard (current population data). Investigate the Multivariate Predictive Power of various groups of Characteristic
Variables NOT included in the Phase 3 Scorecard (current population data).
Go to Recalibration Phase 4 OR Phase 5....
87
Scorecard Recalibration Phase 03h
Go to Recalibration Phase 4
If the Recalibrated Model from Phase 3 is NOT significantly
better than the Phase 2 Recalibrated Model
AND
there are not enough Default Cases for robust model development in
relation to the number of Predictor Variables in the Model.
OR
If the performance of the Recalibrated Model from Phase 3 is NOT
adequate
OR
there are structural problems with the Phase 3 Recalibrated Model,
(insufficient data, time based discontinuities or too few defaults ).8888
Scorecard Recalibration Phase 03h
Go to Recalibration Phase 5
If the Recalibrated Model from Phase 3 is NOT significantly better
than the Phase 2 Recalibrated Model BUT there are sufficient
Default Cases for robust model development.
AND If the performance of the Recalibrated Model from Phase 3 is
NOT adequate OR there are structural problems with the Phase 3
Recalibrated Model, such as insufficient data, time related
discontinuities or known data quality issues.
89
Scorecard Recalibration: Phase 04a
Low Default Portfolios, RELAX the BAD Definition
Investigate the feasibility of relaxing the default definition, from 90 days to 60 days.
Investigate the impact of relaxing the default definition, from 90 days to 60 days.
Investigate the possible segmentation of the portfolio and the corresponding distributions of defaults across and within the various segments.
Apply the Effective Portfolio Sample Size Heuristic to the Portfolio based on the number of defaults and the information contribution of additional Goods in the sample.
Explore Characteristic Variables that may be more appropriate for the relaxed Bad Definition.
90
Scorecard Recalibration Phase 04b
Low Default Portfolios, RELAX the BAD Definition
Investigate alternative Predictor Variables for possible inclusionin the S[60] Scorecard
Univariate Analysis ( Xk: k = 1 to n ) WOE Information Value Variance
Bivariate Analysis Time Series ( Xk , Time ) k = 1 to n Logistic Regression ( Y , Xk ) k = 1 to n
Multivariate Analysis Logistic Regression ( Y , X1 ,X2 ,X3 ,...., Xn) Gini and KS AIC and SBC Interactions / Correlation / Collinearity
91
Scorecard Recalibration Phase 04c
Low Default Portfolios, RELAX the BAD Definition
Select candidate predictor characteristics for inclusion inthe S[60] Scorecard Model:
Variables that are well behaved Statistically
Variables that satisfy the Logistic Regression Model Assumptions
Variables that have Univariate Predictor Power
Variables that have Multivariate Predictor Power
Variables from different Variable Clusters (Risk Factors )
Variables that make good Business Sense ( Behaviour and Portfolio
Transformation)
92
Scorecard Recalibration Phase 04d
Low Default Portfolios, RELAX the BAD Definition
Compare the structure and performance of the two Scorecard
Models S[60] and S[90] based on the respective two Default
Definitions.
Standard Model: S[90]
Relaxed Model: S[60]
Consider a mapping from the S[60] model outcome to the S[90]
model outcome based on the probability : P[Y 90|Y=60 ]
93
Scorecard Recalibration Phase 04e
Go to Recalibration Phase 5
If the Recalibrated Model from Phase 4 is NOT significantly
better than the Phase 3 Recalibrated Model although there are now
sufficient Default Cases for robust model development.
OR
If the performance of the Recalibrated Model from Phase 4 is
NOT adequate
OR
If there are structural problems with the Phase 4 Recalibrated
Model, such as non-compliance with BGH Behaviour Standards
94
Scorecard Recalibration: Phase 5
A: Develop a completely NEW Scorecard Model
B: Compare the NEW Scorecard Model with:
1. The Original Model
2. The Phase 1 Model
3. The Phase 2 Model
4. The Phase 3 Model
5. The Phase 4 Model
95
Scorecard Recalibration Phase 6a
Validation of the Final Logistic Regression Model
1. Model Development Sample : Performance Evaluation
2. Hold-Out Sample : Performance Evaluation
3. Out-OF-Time Sample : Performance Evaluation
4. Data Quality and Model Risk
5. Model Development Methodology
6. Business Requirements
96
Scorecard Recalibration Phase 6b
Implement the Final Recalibrated Model
1. Transformation of the Final Model into a Behaviour Scorecard
2. Behaviour Strategy Development
3. Behaviour Scorecard Approval ( Model, Scorecard, Strategy )
4. Behaviour Scorecard Implementation
5. Behaviour Scorecard Monitoring
6. Portfolio Transformation
97
Scorecard Recalibration Toolkit
With respect to the CRM Scorecard Model Recalibration Toolkit; we have developed a suite of SAS Programs that will enable us to:
Evaluate the present Scorecard relative to the Current Population as opposed to the Development Population [ Quarterly ]
Recalibrate the Existing Scorecard (and its Statistical Model) to the Current Population. ( No modification to the Internal Structure )
Optimise the Internal Attribute Structure of the Characteristics in the present Scorecard relative to the Current Population
Identify weak Explanatory Variables in the present Model which are no longer performing relative to the Current Population or which can no longer be used for business reasons.
Investigate alternative Explanatory Variables which could be used to revitalise the present Scorecard Model by replacing weak Variables.
9898