AML MODEL VALIDATIONA critical need in the new regulatory environment
Frank Masi, Ph.D., EVP Operations
Agenda
• Origins• Model validation (OCC SR11-7)• Regulatory pressure (exit letters, community banks)
– Where, community banks, U.S. wide• Conducting IVV
– Process– Areas
• Reporting• Independence• Vendor Due Diligence• Q&A
© 2015ARC Risk and Compliance
2
3
Model validation has been around for some time.
• Independent Software Verification and Validation (ISVV) derives from the application of IV&V (Independent Verification and Validation) to the software. Early ISVV application (as known today) dates back to the early 1970s when the U.S. Army sponsored the first significant program related to IV&V for the Safeguard Anti-Ballistic Missile System.
• By the end of the 1970s IV&V was rapidly becoming popular. The constant increase in complexity, size and importance of the software lead to an increasing demand on IV&V applied to software (ISVV).
• Meanwhile IV&V (and ISVV for software systems) gets consolidated and is now widely used by organizations such as the DoD, FAA, NASA[1] and ESA.[2] IV&V is mentioned in [DO-178B], [ISO/IEC 12207] and formalized in [IEEE 1012].
• Initially in 2004-2005, a European consortium led by the European Space Agency, and composed by DNV(N),[3] Critical Software SA(P),[4] Terma(DK)[5] and CODA Scisys(UK)[6] created the first version of a guide devoted to ISVV, called "ESA Guide for Independent Verification and Validation" with support from other organizations, e.g. SoftWcare SL (E) ( [7]
), etc.• In 2008 the European Space Agency released a second version, being SoftWcare SL was
the supporting editor having received inputs from many different European Space ISVV stakeholders. This guide covers the methodologies applicable to all the software engineering phases in what concerns ISVV.
© 2015ARC Risk and Compliance
Origins
4
• “Model validation is the set of processes and activities intended to verify that models are performing as expected, in line with their design objectives and business uses. Effective validation helps ensure that models are sound. It also identifies potential limitations and assumptions, and assesses their possible impact.”
(Board of Governors of the Federal Reserve System (SR 11-7), 2011, p. 3)
Purpose of Model Validation
© 2015ARC Risk and Compliance
5
“Model risk occurs primarily for two reasons:
• The model may have fundamental errors and may produce inaccurate outputs when viewed against the design and objective and intended business uses…
• The model may be used incorrectly or inappropriately.”
(Board of Governors of the Federal Reserve System Office of the Comptroller of the Currency (SR 11-7a1), 2011, p. 3)
Model Issues
© 2015ARC Risk and Compliance
6
“All model components—inputs, processing, outputs, and reports—should be subject to validation; this applies equally to models developed in-house and to those purchased from or developed by vendors or consultants.”
(Board of Governors of the Federal Reserve System (SR 11-7), 2011, p. 3)
Model Components
© 2015ARC Risk and Compliance
Inputs Processes Outputs Reports
7
• “Evaluation of Conceptual Soundness. This element involves assessing the quality of the model design and construction, as well as review of documentation and empirical evidence supporting the methods used and variables selected for the model. This step in validation should ensure that judgment exercised in model design and construction is well informed, carefully considered, and consistent with published research and with sound industry practice.”
Key Elements of a Comprehensive Validation
© 2015ARC Risk and Compliance
8
• “Ongoing Monitoring. This step in validation is done to confirm that the model is appropriately implemented and is being used and performing as intended. It is essential to evaluate whether changes in products, exposures, activities, clients, or market conditions necessitate adjustment, redevelopment, or replacement of the model and to verify that any extension of the model beyond its original scope is valid. Benchmarking can be used in this step to compare a given model’s inputs and outputs to estimates from alternatives.”
Key Elements of a Comprehensive Validation
© 2015ARC Risk and Compliance
• “Outcomes Analysis. This step involves comparing model outputs to corresponding actual outcomes. Back-testing is one form of outcomes analysis that involves the comparison of actual outcomes with model forecasts during a sample time period not used in model development at a frequency that matches the model’s forecast horizon or performance window.”
9
Key Elements of a Comprehensive Validation
© 2015ARC Risk and Compliance
Regulatory Pressure
What form we are seeing this:• Exit letters (MRAs, MRIAs)• RFP/RFIs• Service solicitations
Where we are seeing this:
10
Where Who
NY City NJ Foreign Banks
Philadelphia Chicago Wholesale Banks
Ohio Miami Community Banks
Kansas City Tampa Trust Companies© 2015
ARC Risk and Compliance
11
CONDUCTING AN IVV
© 2015ARC Risk and Compliance
12
Parts of a Model Validation
IVV/Model Validation
Documentation Review
Policies
-Risk Assessment-BSA/AML and OFAC Policies-Monitoring Scenarios
Procedures
-Case Review-RFI-SAR Filing
BRD
-Data Mapping-Parsing-Controls
System Review
Input
-Data Mapping-Translated Data-Data Completeness-Data Accuracy-Truncation & Formatting
Process
-Baseline-Rules/Profiles-Configuration-Populations-Statistics
Output
-Alerts\Cases-Logs-Verification
Reporting
-Clear Identification Risks-False Positive Management
© 2015ARC Risk and Compliance
13
A Little Statistics
© 2015ARC Risk and Compliance
Sample Project
14
Week 1
Documentation Review•Review mapping documents;•Review branch risk assessment;•Audit reports;•AML/OFAC policies and procedures;•Product/System manuals;•Business requirement documents;•Functional design documents; and•Previous IVVs.
Week 2
Input Verification•Create and review data samples;•Review data for consistencies, accuracy, and appropriateness;•Controls;•Data Normalization;•Data standardizations;•Reconciliations;
•Data translations;•Data validation (mandatory, required, and supplemental);
Week 3
Process Verification•Review of aggregations, calculations, translations, thresholds and transformations;
•Statistical evidence validating thresholds, parameters, and categories;
Week 4
Output Verification•Demonstrable workflows and reviews;•Effective management reporting;•GAPs between AML policies and procedures;•Management of type 1 and type 2 errors;
Week 5-6
Report•Executive summary;•Top recommendations;•Demonstrate Policies and Procedure to System GAPs;•Document data analysis;•Document product analysis; and•Provide Observations and Recommendations.
© 2015ARC Risk and Compliance
Document Review (examples)
• Review mapping documents;• Review risk assessment;• Audit reports;• AML/OFAC policies and procedures;• Product/System manuals;• Business requirement documents;• Functional design documents; and • Previous IVVs.
15© 2015
ARC Risk and Compliance
Input Review (examples)
• Create and review data samples (focused sampling);• Review data for consistencies, accuracy, and
appropriateness (% populated, column confidence); • Data validation (mandatory, required, and
supplemental);• Review Controls (weak, medium, strong);• Data Normalization (convert to US $);• Data standardizations (US or European date formats);• Reconciliations (all transactions from source are
received and verified);
16© 2015
ARC Risk and Compliance
Process Review (examples)
• Data translations;– Data translations consist of a review and validation are
date format changes, text to currency, address parsing, calculated data, derived data, or any data manipulations within the software.
• Review of aggregations, calculations, translations, thresholds and transformations;
• Reports Analysis
17
0
200000000
400000000
© 2015ARC Risk and Compliance
Process Review (examples)
18
Def-Class (Ex-
empt) 72
HighRisk 26
LowRisk
26
MediumRisk
30
Customers by Risk Class
For-eign
Banks 82
US Cor-pora-tion 28
US Bank 22
For-eign Cor-pora-tion 11
Foreign Owned US Corporation 10
Central Bank 1
Customers by Type
59
41
663 32
1 1 111 1
Customer Distribution by Country of Location
United States of America
CHINA
UNITED KINGDOM
HONG KONG
AUSTRALIA
SINGAPORE
KOREA, REPUBLIC OF
MACAO
PANAMA
JAPAN
BELGIUM
GERMANY
VIET NAM
© 2015ARC Risk and Compliance
Process Review (examples)
19
Both Countries popu-lated2%
Missing Bene or Orig-inator Country
98%
Debit & Credit Country CodesIn All Wires Since Inception
Originator Country Codes Populated
16%
Origina-tor Coun-try Codes Missing
84%
Missing Originator Country CodesIn All Wires Since Inception
MT LC HR2 HR3 LR LLC BOX CE
Baseline 29 18 7 13 15 8 8 1
-20% 31 23 8 16 16 8 10 1
-40% 38 30 8 21 18 11 12 1
-60% 46 47 9 37 23 12 13 1
2.5
12.5
22.5
32.5
42.5
29
18
7
13 15
8 8
1
31
23
8
16 16
8 10
1
38
30
8
2118
11 12
1
46 47
9
37
23
12 13
1
2014 4th Quarter Alerts (Originator)(Baseline and Adjusted)
Baseline -20% -40% -60%
GB 1GU 4
HK 79 KH 191 SG 1
TW 2347
US 1405
VN 377
WT 1
2014 Transactions by Beneficiary Country
GB GU HK KH SG TW US VN WT
© 2015ARC Risk and Compliance
Output Review (examples)
• Demonstrable workflows and reviews;• Effective management reporting;• GAPs between policies and procedures; and• Management of type 1 and type 2 errors.
20
Type 1 Errors (False
Positives)
Type 2 Errors (False
Negatives)
© 2015ARC Risk and Compliance
21
VENDOR DUE DILIGENCE
© 2015ARC Risk and Compliance
Independence
Independence is measured by two factors:• Distance – how far you are removed from the original
project/model setup/changes.• Time – how long since the vendor was involved in the
project/model setup/changes.
A good rule of thumb is that the reviewer should not have been involved in the last setup/changes/review within the last 12 to 18 months.
22© 2015
ARC Risk and Compliance
23
• Independence• Knowledgeable about product or technology• Knowledgeable about compliance• Knowledgeable about process• Knowledgeable about business• Strong model validation methodology
Vendor Qualifiers
© 2015ARC Risk and Compliance
24
A strong model validation policy supports a strong governance program.
A strong model validation policy is risk mitigation.
Conclusion
© 2015ARC Risk and Compliance
References
Federal Financial Institutions Examination Council (2010). Bank Secrecy Act/ Anti-Money Laundering Examination Manual. Retrieved June 21, 2013: http://www.ffiec.gov/bsa_aml_infobase/documents/BSA_AML_Man_2010.pdf.
Board of Governors of the Federal Reserve System Office of the Comptroller of the Currency. (2011, April 4). Supervision and Regulation Letters (SR 11-7a1). Retrieved May 1, 2014, from Board of Governors of the Federal Reserve System: http://www.federalreserve.gov/bankinforeg/srletters/sr1107a1.pdf
Board of Governors of the Federal Reserve System. (2011, April 4). Supervision and Regulation Letters (SR 11-7). Retrieved April 30, 2014, from Board of Governors of the Federal Reserve System: http://www.federalreserve.gov/bankinforeg/srletters/sr1107.htm
25© 2015
ARC Risk and Compliance
THANK YOU
26
Questions
Contact Information:
Frank Masi, [email protected] ext. 102 http://www.arcriskandcompliance.com
© 2015ARC Risk and Compliance
Top Related