Making Models Work for You
Sound model development requires an understanding of what a model is while also understanding situations that warrant the use of the model. Additionally, you must understand relevant model approaches and be able to differentiate a good model from a bad one. Once understood, the modeling approach must consist of several critical steps to ensure creation of a reasonable model that is based upon rigorous analysis.
WithumSmith+Brown provides model validation and model creation services through its Forensic & Valuation Services group.
What is a Model (SMUUR)?
Before a model can be developed, you first must understand what a model actually is. It is quite common for someone to be unable to accurately communicate what a model is or does, so we turn to the dictionary for an exact definition. A model is defined by Oxford English dictionary as, “a simplified description, especially a mathematical one, of a system or process, to assist calculations and predictions.” Well what does this actually mean? A model is:
- Simplification – a model is a simplification of the real world and takes a seemingly complex problem and creates a parsimonious and hopefully understandable representation. All models follow a fairly simple structure as is presented below, but this structure is important nonetheless;
- Mathematical – A model typically consists of equations or formulas though this is not always the case;
- Uncertain – The output from a model is not absolute and can vary given different assumptions (e.g. 2+2 and Sales minus cost of goods sold are not models and are merely calculations);
- Useful – The model must assist the user in the decision-making process; and
- Repeatable – If different users enter the same information into the model, they must retrieve the same results.
Why do we need a model?
With a model defined, it is easy to turn to situations where a model would be necessary or beneficial. In our personal lives, we could use a model to determine whether we should buy or lease a car or whether we should lend our best friend money. What are the odds the Mets make the World Series ? Professionally, we may need a model to determine the value of a pharmaceutical company or forecast how much employment will increase next month. All of these examples illustrate why a model is warranted because a model:
- Takes a seemingly complex problem and simplifies it into a procedure where data is processed and output is produced that is:
- understandable,
- reasonable,
- defendable,
- quantified (e.g. rank ordering, probability, coefficients/correlations, scores),
- testable,
- repeatable, and
- actionable;
- Allows for different scenarios or a sensitivity analysis;
- Produces results that are easily transferable if the modeler “wins the lottery”; and
- Is oftentimes required by regulating authorities (e.g. SEC or IRS).
Common Modeling Approaches
The development of a model will more than likely fall under one of three approaches differentiated by their reliance on inputs. The three approaches are described below.
Subjective or Qualitative Approach
This approach relies on the input of experts or committees and/or anecdotal information.
ADVANTAGES | DISADVANTAGES |
It requires little if any data and can consider factors not captured in the data. Additionally, it can be easier to explain and comprehend. | It is likely to be subjective and will be difficult to support and test. Additionally, a subjective model can be difficult to replicate and benchmark as an “expert’s” biases can change over time. |
Quantitative or Statistical Approach
This approach relies exclusively on hard data and statistical and mathematical processes.
ADVANTAGES | DISADVANTAGES |
It is fully objective as only data drives the output of the model, a clear relationship is defined between the explanatory and explained variables and the model is testable, replicable and able to be benchmarked through time. | It ignores factors not captured in the data and can be difficult to explain and comprehend for people with limited quantitative backgrounds. Additionally the model is only as predictive as the data loaded into the model. |
The Hybrid Approach
This approach synthesizes the other two approaches to focus on the positives and limit the negatives of each approach. This approach typically begins with a quantitative model but allows an expert to adjust the output of said model for other factors (ex. a forecast of U.S. gross domestic product or GDP begins with an underlying statistical model but allows for an economist to make adjustments to the model output).
ADVANTAGES | DISADVANTAGES |
It allows the expert to account for factors not captured in the data while still maintaining all of the advantages of a statistical model (i.e. objective, replicable, testable, etc.) | It requires both a quantitative model and an “expert” with an intimate knowledge of the model and problem. Additionally, “expert” biases can result in over or under adjustments. |
General Modeling Approach
Model development requires six general steps be performed in a particular order to ensure common mistakes and pitfalls our avoided. The following outlines these steps and points out things that should be occurring in these steps:
- Research the problem (e.g. scholarly articles, experts, colleagues) for potential explanatory factors and modeling approaches.
- Get buy in from key stakeholders on hypothesis and planned approach(es)
- Synthesize hypothesis and document planned approach accordingly
Define the Problem the Model will help to Solve |
|
Generate a Hypothesis |
Collect All Necessary Data |
|
Produce the Model |
|
Test the Model |
|
Implement Model and Act on Model Results |
|
It is important to note that data collection is the third step of the process. This is imperative as you run into numerous issues (notably data mining) if you gather data before clearly defining the problem and hypothesizing about the model factors. You must understand the issue at hand and probable solution before collecting the appropriate data.
The model pitfall is but one of many model pitfalls that can occur. The table below lists additional pitfalls and resulting consequences though this list is not all inclusive.
Pitfall | Consequence |
---|---|
Incorrectly defined problem and/or model need | Output doesn’t address the problem at hand |
Collecting data/building a model without generating a hypothesis | Unfounded model structure based upon data mining |
Bad or inappropriate data | Model not useful (GIGO: Garbage in, Garbage out) |
Not getting appropriate approvals | Stakeholders may reject approach |
Inadequate documentation | Model not replicable and transferable |
Limitations not understood | Incorrect conclusions reached |
Model unnecessarily complex | Users may not understand and difficult to monitor |
Inadequate testing | Model misspecification |
Not implementing the appropriate controls | User error not caught |
No model monitoring | Model may become stale |
Conclusion
With a better understanding of how a model is defined and an outline for a general modeling approach, you are better prepared to avoid common mistakes and pitfalls and create more reliable and reasonable models. It is imperative to follow the steps presented above exactly all the while asking yourself these important questions:
- Who is the final user of my model and what is their sophistication level?
- What are the external inputs and assumptions of the model and how will the model need to process them (e.g. flexible for scenario analysis?)
- What are the limitations of the proposed approach?
- What is the best platform for the planned model?
- How often will the model or assumptions need to be “refreshed”?
- How will the model need to be reviewed or monitored?
- What is the required output of the model?
By following the outlined modeling approach and taking the time to understand the problem, proposed model, and model users, the results of you modeling endeavor will undeniably be more useful and better prepared to weather the rigors of any review.
Jason Kunkel 609-520-1188 [email protected] |
Ask Our Experts
To ensure compliance with U.S. Treasury rules, unless expressly stated otherwise, any U.S. tax advice contained in this communication is not intended or written to be used, and cannot be used, by the recipient for the purpose of avoiding penalties that may be imposed under the Internal Revenue Code.