Dissonant Models and Distracting Measures?

At Awbury, we try to avoid being caught out by “framing” issues, in which the use of or adherence to particular cognitive pathways can lead to “blind spots” when analyzing or assessing risks.

It is axiomatic that measuring credit risk is about trying to identify what the most important ones are for a particular obligor, portfolio or scenario, and then assigning probabilities to one or more of them causing distress or default.

Problems arise when there is a lack of data on relevant past events, coupled with types of risk that are infrequent, such as systemic financial crises. We know they occur; but because they are infrequent predicting them and their outcomes can be a futile exercise.

A good example of this is the point made by the research foundation, Vox, in a paper entitled “The dissonance of the short and long term”- that an OECD member country suffers a crisis every 43 years on average; while true global financial crises are even less frequent- consider the time period between the Great Depression and the Great Recession. So, if modern financial markets are not even 200 years old, the sample size available for predictive purposes is very small.

Of course, models such as Monte Carlo simulations are supposed to be able to tease out the extremes of possible distributions. However, they are only models, and not representations of the real world. As actual experience during the Great Financial Crisis amply demonstrated, events that (according to the then-existing models) are not supposed to be able to happen during the known life of the Universe nevertheless do, because the predictive models which told one that was impossible were deeply flawed.

Another problem with risk measurement is that if people believe that they can track, measure and model a particular risk factor, they may tend to focus on it, because it can be measured; and so fall prey to being caught within a frame of reference. As a result, by focusing on short term, measurable factors, they overlook or ignore the more important and potentially threatening longer term ones. For example, economists are constantly seeking to measure the various factors which they believe are harbingers of recession (or future trends in interest rates), yet it is truism that they generally would be better tossing a coin, because recessions (or movements in interest rates) are the result of the interplay of multiple complex factors, many of which are not (at least yet) truly measurable.

As Goodhart’s Law states: when a measure becomes a target, it is subsequently no longer a good measure. For example, if people anticipate the effect of a policy, and their actions therefore alter the policy’s outcome, the target was a bad measure. In other words, measuring isolated factors is a distraction from careful and thoughtful analysis. The “beauty” of the models is too alluring.

Using short term measurements to drive complex decisions with long term outcomes is simply foolish.

From Awbury’s point of view, while we, of course, use multiple types of models as part of our risk analysis and management process, we aim to avoid becoming seduced by their apparent certainty; always overlaying their outputs with an element of robust “but what if we’re wrong?” and “what might we have missed?” thought experiments- our “testing a thesis to destruction” approach.

The Awbury Team


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.