Making Models Work for You

Making Models Work for You

Share on LinkedIn
Share on Facebook
Tweet Me
Subscribe to Withum News
It is difficult to find someone who does not use models in their personal or professional lives on a regular basis. To avoid common mistakes and pitfalls arising in the development or use of models, it is imperative that the process by which a model is developed and implemented follows several key steps.

Sound model development requires an understanding of what a model is while also understanding situations that warrant the use of the model. Additionally, you must understand relevant model approaches and be able to differentiate a good model from a bad one. Once understood, the modeling approach must consist of several critical steps to ensure creation of a reasonable model that is based upon rigorous analysis.

WithumSmith+Brown provides model validation and model creation services through its Forensic & Valuation Services group.

What is a Model (SMUUR)?

Before a model can be developed, you first must understand what a model actually is. It is quite common for someone to be unable to accurately communicate what a model is or does, so we turn to the dictionary for an exact definition. A model is defined by Oxford English dictionary as, “a simplified description, especially a mathematical one, of a system or process, to assist calculations and predictions.” Well what does this actually mean? A model is:

  • Simplification – a model is a simplification of the real world and takes a seemingly complex problem and creates a parsimonious and hopefully understandable representation. All models follow a fairly simple structure as is presented below, but this structure is important nonetheless;chart
  • Mathematical – A model typically consists of equations or formulas though this is not always the case;
  • Uncertain – The output from a model is not absolute and can vary given different assumptions (e.g. 2+2 and Sales minus cost of goods sold are not models and are merely calculations);
  • Useful – The model must assist the user in the decision-making process; and
  • Repeatable – If different users enter the same information into the model, they must retrieve the same results.

Why do we need a model?

With a model defined, it is easy to turn to situations where a model would be necessary or beneficial. In our personal lives, we could use a model to determine whether we should buy or lease a car or whether we should lend our best friend money. What are the odds the Mets make the World Series ? Professionally, we may need a model to determine the value of a pharmaceutical company or forecast how much employment will increase next month. All of these examples illustrate why a model is warranted because a model:

  • Takes a seemingly complex problem and simplifies it into a procedure where data is processed and output is produced that is:
    • understandable,
    • reasonable,
    • defendable,
    • quantified (e.g. rank ordering, probability, coefficients/correlations, scores),
    • testable,
    • repeatable, and
    • actionable;
  • Allows for different scenarios or a sensitivity analysis;
  • Produces results that are easily transferable if the modeler “wins the lottery”; and
  • Is oftentimes required by regulating authorities (e.g. SEC or IRS).

Common Modeling Approaches

The development of a model will more than likely fall under one of three approaches differentiated by their reliance on inputs. The three approaches are described below.

Subjective or Qualitative Approach

This approach relies on the input of experts or committees and/or anecdotal information.

ADVANTAGES DISADVANTAGES
It requires little if any data and can consider factors not captured in the data. Additionally, it can be easier to explain and comprehend. It is likely to be subjective and will be difficult to support and test. Additionally, a subjective model can be difficult to replicate and benchmark as an “expert’s” biases can change over time.

Quantitative or Statistical Approach

This approach relies exclusively on hard data and statistical and mathematical processes.

ADVANTAGES DISADVANTAGES
It is fully objective as only data drives the output of the model, a clear relationship is defined between the explanatory and explained variables and the model is testable, replicable and able to be benchmarked through time. It ignores factors not captured in the data and can be difficult to explain and comprehend for people with limited quantitative backgrounds. Additionally the model is only as predictive as the data loaded into the model.

The Hybrid Approach

This approach synthesizes the other two approaches to focus on the positives and limit the negatives of each approach. This approach typically begins with a quantitative model but allows an expert to adjust the output of said model for other factors (ex. a forecast of U.S. gross domestic product or GDP begins with an underlying statistical model but allows for an economist to make adjustments to the model output).

ADVANTAGES DISADVANTAGES
It allows the expert to account for factors not captured in the data while still maintaining all of the advantages of a statistical model (i.e. objective, replicable, testable, etc.) It requires both a quantitative model and an “expert” with an intimate knowledge of the model and problem. Additionally, “expert” biases can result in over or under adjustments.

General Modeling Approach

Model development requires six general steps be performed in a particular order to ensure common mistakes and pitfalls our avoided. The following outlines these steps and points out things that should be occurring in these steps:

  • Research the problem (e.g. scholarly articles, experts, colleagues) for potential explanatory factors and modeling approaches.
  • Get buy in from key stakeholders on hypothesis and planned approach(es)
  • Synthesize hypothesis and document planned approach accordingly
Define the Problem the Model will help to Solve
  • Work to understand the issue at hand that requires a model
  • Get key stakeholders on same page and agree upon a clearly defined description of the problem
  • Understand the desired output of the model as it relates to the problem
  • Codify problem and document appropriately
Generate a Hypothesis
Collect All Necessary Data
  • Define reputable sources for all external data
  • Collect and database necessary internal and external data
  • Analyze data for anomalies, errors, and outliers
  • Perform any necessary data adjustments
  • Document and get buy in from key stakeholders all sources used, data collected, and exclusions and adjustments made
Produce the Model
  • Determine the appropriate platform with which to build your model
  • Based upon hypothesized approach and data collected, develop model or models for further testing
  • Present modeling results to key stakeholders and solicit feedback
  • Document modeling approach and model results
Test the Model
  • Preform any and all necessary model tests
  • If appropriate, perform scenario and sensitivity analyses
  • If multiple models developed, compare results and rank
  • Peer review
  • Develop ongoing model monitoring plan
  • Document tests performed and testing results noting all model limitations
Implement Model and Act on Model Results
  • Present result of all previous steps to key stakeholders and get approval of final model
  • Determine best platform for model dissemination
  • Implement appropriate model controls
  • Train intended users of the model and create user manual

It is important to note that data collection is the third step of the process. This is imperative as you run into numerous issues (notably data mining) if you gather data before clearly defining the problem and hypothesizing about the model factors. You must understand the issue at hand and probable solution before collecting the appropriate data.

The model pitfall is but one of many model pitfalls that can occur. The table below lists additional pitfalls and resulting consequences though this list is not all inclusive.

Pitfall Consequence
Incorrectly defined problem and/or model need Output doesn’t address the problem at hand
Collecting data/building a model without generating a hypothesis Unfounded model structure based upon data mining
Bad or inappropriate data Model not useful (GIGO: Garbage in, Garbage out)
Not getting appropriate approvals Stakeholders may reject approach
Inadequate documentation Model not replicable and transferable
Limitations not understood Incorrect conclusions reached
Model unnecessarily complex Users may not understand and difficult to monitor
Inadequate testing Model misspecification
Not implementing the appropriate controls User error not caught
No model monitoring Model may become stale

Conclusion

With a better understanding of how a model is defined and an outline for a general modeling approach, you are better prepared to avoid common mistakes and pitfalls and create more reliable and reasonable models. It is imperative to follow the steps presented above exactly all the while asking yourself these important questions:

  • Who is the final user of my model and what is their sophistication level?
  • What are the external inputs and assumptions of the model and how will the model need to process them (e.g. flexible for scenario analysis?)
  • What are the limitations of the proposed approach?
  • What is the best platform for the planned model?
  • How often will the model or assumptions need to be “refreshed”?
  • How will the model need to be reviewed or monitored?
  • What is the required output of the model?

By following the outlined modeling approach and taking the time to understand the problem, proposed model, and model users, the results of you modeling endeavor will undeniably be more useful and better prepared to weather the rigors of any review.

Jason Kunkel Jason Kunkel
609-520-1188
[email protected]

linkedin

Ask Our Experts

To ensure compliance with U.S. Treasury rules, unless expressly stated otherwise, any U.S. tax advice contained in this communication is not intended or written to be used, and cannot be used, by the recipient for the purpose of avoiding penalties that may be imposed under the Internal Revenue Code.

Previous Post

Next Post