Sound model development requires an understanding of what a model is while also understanding situations that warrant the use of the model. Additionally, you must understand relevant model approaches and be able to differentiate a good model from a bad one. Once understood, the modeling approach must consist of several critical steps to ensure creation of a reasonable model that is based upon rigorous analysis.
WithumSmith+Brown provides model validation and model creation services through its Forensic & Valuation Services group.
Before a model can be developed, you first must understand what a model actually is. It is quite common for someone to be unable to accurately communicate what a model is or does, so we turn to the dictionary for an exact definition. A model is defined by Oxford English dictionary as, “a simplified description, especially a mathematical one, of a system or process, to assist calculations and predictions.” Well what does this actually mean? A model is:
With a model defined, it is easy to turn to situations where a model would be necessary or beneficial. In our personal lives, we could use a model to determine whether we should buy or lease a car or whether we should lend our best friend money. What are the odds the Mets make the World Series ? Professionally, we may need a model to determine the value of a pharmaceutical company or forecast how much employment will increase next month. All of these examples illustrate why a model is warranted because a model:
- Takes a seemingly complex problem and simplifies it into a procedure where data is processed and output is produced that is:
- quantified (e.g. rank ordering, probability, coefficients/correlations, scores),
- repeatable, and
- Allows for different scenarios or a sensitivity analysis;
- Produces results that are easily transferable if the modeler “wins the lottery”; and
- Is oftentimes required by regulating authorities (e.g. SEC or IRS).
The development of a model will more than likely fall under one of three approaches differentiated by their reliance on inputs. The three approaches are described below.
This approach relies on the input of experts or committees and/or anecdotal information.
|It requires little if any data and can consider factors not captured in the data. Additionally, it can be easier to explain and comprehend.||It is likely to be subjective and will be difficult to support and test. Additionally, a subjective model can be difficult to replicate and benchmark as an “expert’s” biases can change over time.|
This approach relies exclusively on hard data and statistical and mathematical processes.
|It is fully objective as only data drives the output of the model, a clear relationship is defined between the explanatory and explained variables and the model is testable, replicable and able to be benchmarked through time.||It ignores factors not captured in the data and can be difficult to explain and comprehend for people with limited quantitative backgrounds. Additionally the model is only as predictive as the data loaded into the model.|
This approach synthesizes the other two approaches to focus on the positives and limit the negatives of each approach. This approach typically begins with a quantitative model but allows an expert to adjust the output of said model for other factors (ex. a forecast of U.S. gross domestic product or GDP begins with an underlying statistical model but allows for an economist to make adjustments to the model output).
|It allows the expert to account for factors not captured in the data while still maintaining all of the advantages of a statistical model (i.e. objective, replicable, testable, etc.)||It requires both a quantitative model and an “expert” with an intimate knowledge of the model and problem. Additionally, “expert” biases can result in over or under adjustments.|
Model development requires six general steps be performed in a particular order to ensure common mistakes and pitfalls our avoided. The following outlines these steps and points out things that should be occurring in these steps:
|Define the Problem the Model will help to Solve|
|Generate a Hypothesis|
|Collect All Necessary Data|
|Produce the Model|
|Test the Model|
|Implement Model and Act on Model Results|
It is important to note that data collection is the third step of the process. This is imperative as you run into numerous issues (notably data mining) if you gather data before clearly defining the problem and hypothesizing about the model factors. You must understand the issue at hand and probable solution before collecting the appropriate data.
The model pitfall is but one of many model pitfalls that can occur. The table below lists additional pitfalls and resulting consequences though this list is not all inclusive.
|Incorrectly defined problem and/or model need||Output doesn’t address the problem at hand|
|Collecting data/building a model without generating a hypothesis||Unfounded model structure based upon data mining|
|Bad or inappropriate data||Model not useful (GIGO: Garbage in, Garbage out)|
|Not getting appropriate approvals||Stakeholders may reject approach|
|Inadequate documentation||Model not replicable and transferable|
|Limitations not understood||Incorrect conclusions reached|
|Model unnecessarily complex||Users may not understand and difficult to monitor|
|Inadequate testing||Model misspecification|
|Not implementing the appropriate controls||User error not caught|
|No model monitoring||Model may become stale|
With a better understanding of how a model is defined and an outline for a general modeling approach, you are better prepared to avoid common mistakes and pitfalls and create more reliable and reasonable models. It is imperative to follow the steps presented above exactly all the while asking yourself these important questions:
By following the outlined modeling approach and taking the time to understand the problem, proposed model, and model users, the results of you modeling endeavor will undeniably be more useful and better prepared to weather the rigors of any review.
To ensure compliance with U.S. Treasury rules, unless expressly stated otherwise, any U.S. tax advice contained in this communication is not intended or written to be used, and cannot be used, by the recipient for the purpose of avoiding penalties that may be imposed under the Internal Revenue Code.