Model Validation Process
The following model validation process applies to financial models and particularly to risk, scorecard, and portfolio analysis models. ALI deploys the below check list for a thorough validation process.
- Has there been an adequate literature review for this type of model done by the modeler whether is if POD, Loss Forecast, or Risk model.
- Did the modeler refer to past models for which model may already be deployed in production. The existing model been benchmarked
- Was the model methodology researched and vetted. There are a number of techniques such as for POD logistic, Decision Tree, SVM, GBM, Neural Network
- Have alternative theories been considered such as new technology or approaches for this type of models.
- Model validation documentation has been thorough and ensured completeness of all the steps involved in the process including all the results been listed so intermediate steps and results can be referred back to each step. This document become extremely useful for future development of the same model. Such as:
- What are the model assumptions
- Are the model assumptions valid
- Have data quality checks been conducted
- Is the variable selection process documented
- Did the modeler document variable transformations such as binning of continuous variables into categorical variables or segmentation of discrete variables into fewer levels? All these transformation will need to be conducted during deployment phase.
- Has the model developer performed back testing with out of sample or out of time samples?
- Was benchmark analysis performed by the model developer?
- Expert advice been documented for complete accountability such as minutes of meetings and emails kept.
- Documentation of model approval group
- Model implementation documented along with implementation specs that are intelligible for the implementation team.
- Model maintenance and monitoring plan documented? Weekly, Monthly, Quarterly.
- Have business requirements documented
- Have the model outputs explained adequately
- Dose the model comply with all applicable regulations
- Model version control is maintained
- Consequences of model assumption violations listed by modeler.
- Model dependencies clearly explained. If outputs of one model are inputs for another model.
- Model Development documented
- Data definitions (dictionary, every variable explained granularly)
- Data source details both internal and external
- Data segmentation
- Data Cleaning
- Sampling Details
- Missing Values Details. If any values were imputed and how outliers were treated
- Data Lineage Documented such as from trading floor and subsequent updates.
- Segmentation strategy and its rationale provided
- Detailed variable selection criteria – initial variables, reduction criteria – KS, Gini, IV, etc
- Mathematical descriptions
- Diagnostic checks and test results well documented
- Model source code and development data should be properly kept with proper version control with proper access control. Final development data kept in the same location.
- Stress testing and scenario testing
- Sensitivity analysis to certain change in values of variables or shocks
- Roles and responsibilities. Model owner, model monitor, implementer, project manager.
- Model risk and limitations whether it is theoretical or data issues should be listed. The overall model risk to the business listed and quantified. Cross functional teams involved i.e., Business, IT, Model Development team.
- Implementation details – key person exposure, computational time, definition changes, etc.
- UAT – Testing and documentation of each detail
ALI analytics Edge will:
- Provide comprehensive external validation for your institution’s A, B and C level models.
- We will do this in a cost effective and timely manner reducing your need for additional resources.
- Improve your regulatory compliance, improve governance of models and enhance stress tests.
- Act with confidence that we will provide an unbiased look at your models and bring in an external perspective to your models making them more robust.
Contact ALI today to learn how our model validation experts can help your organization be more effective in meeting mission goals. We help you do more with less.
Power of Analytics for your Analytics edge with ALI superior independent validation of all your models