Model Risk Overview. Framework for model risk management

1. © 2018 RSM US LLP. All Rights Reserved.© 2018 RSM US LLP. All Rights Reserved.

2. © 2018 RSM US LLP. All Rights Reserved.© 2018 RSM US LLP. All Rights Reserved.
MODEL RISK MANAGEMENT
2018 Bank Director’s Assembly
March 5, 2018

3. © 2018 RSM US LLP. All Rights Reserved.
Agenda
• What are Models?
• Model regulatory considerations
• Model usage
• What is Model Risk?
• Model Risk Management
• Model classification
• Model back-testing/validation
• Model development/procurement
• DFAST & CECL
• Audit considerations
• Questions – 5 minutes
3

4. © 2018 RSM US LLP. All Rights Reserved.
What are Models?
• Simplified representations of real world relationships among observed
characteristics, values and events
• Quantitative methods, system or approach that applies statistical,
economic, financial or mathematical theories, techniques and assumptions
to process input data into quantitative estimates
• Quantitative approaches whose inputs are partially or wholly qualitative or
based on expert judgment provided the output is quantitative
Information Input
Data
Assumptions
Processing Component
Transforms input into
estimates
Reporting component
Translates estimates
into useful business
information
4

5. © 2018 RSM US LLP. All Rights Reserved.
Model Regulatory Considerations
• Trading and Capital Markets Activities Manual (FED)
• Supervisory Guidance on Model Risk Management (OCC Bulletin 2011-12)
− Originally covered model validation. Expanded in 2011 to cover Model Risk
Management and governance
• Guidance on Model Risk Management SR 11-7 (FED)
• Comptroller’s Handbook July 2016 – Management & Asset Quality (OCC)
• Bank Holding Supervision Manual Section 2126 – Model Risk Management -
Sept 2017 (FED)
• Vendor Models for Credit Risk Management (BCBS)
• Stress testing for banks $10B to $50B SR 14-3 (FED)
• Supervisory Assessment of Capital Planning and Positions for LISCC Firms and
Large and Complex Firms SR 15-18 (FED)
• Supervisory Assessment of Capital Planning and Positions for Large and
Noncomplex Firms SR 15-19 (FED)
• 2018 Supervisory Scenarios for Annual DFAST Rules (FED)
5

6. © 2018 RSM US LLP. All Rights Reserved.
Where are Models used?
• Valuing exposure
− Exposure
− Instruments
− Positions
• Pricing models
− Value and hedge complex
securities
• Underwriting credits
• Managing and safeguarding
customer assets
• Determining capital and reserve
adequacy
• Collections
• Stress testing
• ALLL
• Anti- money laundering
• Fraud models
• Measuring risk
• Managing and safeguarding
customer assets
• Analyzing business strategies
• Credit scoring
• Behavior scoring
• Collections
• Informing business decisions
• Identifying and measuring risks
• Issuing public disclosures
• Meeting regulatory requirements
6

7. © 2018 RSM US LLP. All Rights Reserved.
Typical Financial Models
Exposure
at
Default (EAD)
Loss Given Default
(LGD)
Probability of Default (PD)
The less common the model – the more dependency on good model risk
management processes
Most common model
• Probability of a credit event –
delinquency or default
• Usually independent of specific bank
process
Less common model
• Model of losses once in default
• Somewhat dependent on specific bank
process
Least common model
• Model of level of exposure at default
• Dependent on specific bank process
7

8. © 2018 RSM US LLP. All Rights Reserved.
What is Model Risk?
• Model risk occurs for three primary reasons:
− The input data is faulty.
− The model has fundamental errors.
− The model is used incorrectly – for example, applied to
new products or markets.
• Impact
− Financial loss
− Poor business decisions
− Damage to reputation
Input data is
faulty
Model has
fundamental
errors
Model is used
incorrectly
Potential for adverse consequences from decisions based on
incorrect or misapplied model outputs and reports
8

9. © 2018 RSM US LLP. All Rights Reserved.
What is Model Risk Management?
• Model Inventory
• Model risk ratings
• Model controls
• Model back-
testing
• Models validated
in accordance
with risk rating
• Model validation
managed
• Models
appropriate for
the business
• Models
developed and
documented
• Model Risk
Framework
appropriate for
the business
• Polices,
procedures
• Responsibilities Governance,
Policies and
Controls
Model
Selection &
Development
Model
Management
Model
Validation
9

10. © 2018 RSM US LLP. All Rights Reserved.
Determining what is a Model
10
Models
• Formula(s) and programs for
calculating of value at risk
(VAR). Application of a Monte
Carlo simulation or historical
data simulation
• Linier regression calculation
of the probability of default
(PD)
Not a Model
• Simple mathematical data
manipulations – addition,
subtraction, averages,
variation, ratios
• Growth projections based on
single variable analysis
Model management is complex choose models wisely.
10

11. © 2018 RSM US LLP. All Rights Reserved.
Risk Rating Models – Not all models are created equal
• Impact to financials
• Impact to strategic decisions
• Impact to reputation
High
Medium
Low
• Input data quality
• Model validation
• Documentation maturity
• Products modeled
High
Medium
Low
• New technology / statistics
• Technical to build
• Technical to use
• Number variables
High
Medium
Low
Risk
Exposure
Model Application
Model Complexity
Factor Risk Rating Variables
11

12. © 2018 RSM US LLP. All Rights Reserved.
Source of Model Risk
Information Input
Processing Component
Reporting component
Transforms input into
estimates
Fundamental errors
• Model does not correlate
with reality
• Mathematical
assumptions are flawed
• Model instability &
inconsistencies
• Outdated model
• Unproven statistical basis
• Calculation errors and
errors in model logic
Data and Assumptions
Input data in faulty
• Incorrect data
• Data mapping errors
• Data Quality
• Accurate
• Complete
• Data depth is
insufficient
• Data source is external
& uncontrolled
• Define data proxies
• Assumptions invalid
Translates estimates
into useful business
information
Model is used incorrectly
• Irrelevant and inaccurate
reporting
• Design purpose and use
purpose differ
• Model extended beyond
original purpose
• Unmanaged changes to
model
Like credit risk and market risk, model risk cannot be eliminated.
12

13. © 2018 RSM US LLP. All Rights Reserved.
Model Back-testing – Model Validation
13
Back-testing
• Back-testing - quantitative
assessment of a model
that is based on the
comparison of forecasts
against realized values.
• A back-testing program
refers to the whole
process of conducting
back-testing
Validation
• Validation - the audited
process of “validating” the
functionality and
completeness of a model.
• Review of the logical and
conceptual soundness of
the model.
Both Back-testing and Validation are required for assurance.
13

14. © 2018 RSM US LLP. All Rights Reserved.
Assessing a Vendor Model
Vendor – Internal supplier assessment process
Model Basis
• Geography of population
• Population demographics
• Types of obligors, exposures,
firm size, etc.
Bank
Data Set
Aligns
Model Validation Process
• Minimal technical investment
• Description of methods used
• Clear documentation
• Level of model customization
• Validation of selected model
• Frequency of vendor revalidation
of models
• Ease of validation by Bank
• Ease of use for stress testing
14

15. © 2018 RSM US LLP. All Rights Reserved.
Model Validation - Input assessment
Processing
One Model
with different
inputs
Input 1
Input 2
• Data variation
• Time
• Content
• Product
• Extreme
• Assumption variation
Comparative
1 vs 2
Model Behavior
• Assess Model
• Robust
• Stable
• Accurate
• Limitations
• Document
everything
Reporting 1
Reporting 2
Evaluating the model behavior
over a range of inputs15

16. © 2018 RSM US LLP. All Rights Reserved.
Model Validation – Compare models
Reporting 1Processing 1
• Hold input
constant
• Run model Comparative
1 vs 2
• Assess the
variation of
the reporting
output
• Identify
assignable
causes
• Document
everything
Input
• Hold inputs
constant
• Hold
assumptions
constant
Reporting 2Processing 2
• Hold input
constant
• Run model
Comparing different models with the same data set
16

17. © 2018 RSM US LLP. All Rights Reserved.
Best Practice Model Development
Developers
• Multi-disciplinary, economists, finance, statisticians, mathematics
• Technical knowledge
• Experience
Sound Practice
• Numerical techniques
• Mathematical specifications
• Approximations
Testing and validation
• Rigorous assessment of data quality
• Comparison of alternate theories
• Mathematically and statistically correct
Documentation
• Design, theory, logic
• Test plans and documented results
• Business use documentation
17

18. © 2018 RSM US LLP. All Rights Reserved.
DFAST – Dodd Frank Act Stress Testing Models
Baseline Adverse
Severely
Adverse
Material Risks
• Credit Risks
• Interest Rate Risk
• Liquidity Risk / Price Risk
• Six economic indicators
• Four aggregate measures of asset prices
• Six measures of interest rates
• International – 3 variable in 4 country blocks
Many models aggregated
Individual model risk and aggregated model risk18

19. © 2018 RSM US LLP. All Rights Reserved.© 2018 RSM US LLP. All Rights Reserved.
19
CECL – Current Expected Credit Loss
• Credit losses expected over the life of the loan are
effectively recorded upon origination
• Bankers also are concerned with reliability of the
CECL model -- and the volatility that is the result of
that
• As with other models used for financial decision-
making, a new model used to estimate the ALLL
under CECL will be subject to regulatory model risk
management standards.
American Banker

20. © 2018 RSM US LLP. All Rights Reserved.
Model Risk Audit considerations
20
Governance, Policies and Controls
• Governance framework and organization
• Model Risk management resources:
size, qualifications, independence,
outsourcing, validation
• Business Continuity and disaster
recovery planning
Model Selection & Development
• Accuracy and completeness of model
• Evaluate the reliability of the data input to
the models
• Relevancy of data and information
reported to management
Model Validation
• Validations performed
• Effective challenges to the model
• Assess both individual use and the
aggregated use of models.
Model Management
• Approval and records / documentation
(including emergency model changes)
• Model controls as related to the identified
model weaknesses
• Process for model limits and model
usage
• Back-testing / Stress testing
Goal – overall effectiveness of policies and compliance

21. © 2018 RSM US LLP. All Rights Reserved.
Summary – Agenda topics
21
• What are Models?
• Model regulatory considerations
• Model usage
• What is Model Risk?
• Model Risk Management
• Model classification
• Model back-testing/validation
• Model development/procurement
• DFAST & CECL
• Audit considerations

22. © 2018 RSM US LLP. All Rights Reserved.© 2018 RSM US LLP. All Rights Reserved.
22

23. © 2018 RSM US LLP. All Rights Reserved.© 2018 RSM US LLP. All Rights Reserved.
23

24. © 2018 RSM US LLP. All Rights Reserved.
This document contains general information, may be based on authorities that are subject to change, and is not a substitute for professional
advice or services. This document does not constitute audit, tax, consulting, business, financial, investment, legal or other professional
advice, and you should consult a qualified professional advisor before taking any action based on the information herein. RSM US LLP, its
affiliates and related entities are not responsible for any loss resulting from or relating to reliance on this document by any person. Internal
Revenue Service rules require us to inform you that this communication may be deemed a solicitation to provide tax services. This
communication is being sent to individuals who have subscribed to receive it or who we believe would have an interest in the topics
discussed.
RSM US LLP is a limited liability partnership and the U.S. member firm of RSM International, a global network of independent audit, tax and
consulting firms. The member firms of RSM International collaborate to provide services to global clients, but are separate and distinct legal
entities that cannot obligate each other. Each member firm is responsible only for its own acts and omissions, and not those of any other
party. Visit rsmus.com/aboutus for more information regarding RSM US LLP and RSM International.
RSM® and the RSM logo are registered trademarks of RSM International Association. The power of being understood® is a registered
trademark of RSM US LLP.
© 2018 RSM US LLP. All Rights Reserved.
RSM US LLP
+1 800 274 3978
www.rsmus.com