Beware Forecasts!

[Kirkhill]

I have been a professional mathematical modeller for the past 40 years. Over that period my confidence in models’ forecasts, my own included, has fallen and fallen.

I now think that models of complex systems are of little or no use to decision-makers – they are subject to too much bias, too many spurious correlations. They endow forecasts with an aura of ‘expertise’ that is, in my view, misplaced. Beware forecasts.

Some of the best predictive models turn out to be simple (with perhaps only one or two variables included), even though the historical “fit” is not that good.

Good predictive power tends to come where the modellers are genuinely on top of the theory as well as the data. So, for example, modellers of airflow over an aircraft wing will build a model to predict what angle and speed combination will induce a stall in an aircraft, and these models are amazingly reliable (thank goodness).

But that’s because the modellers really understand the limited influences on airflow over a wing, and include only those variables which are genuinely relevant. This is not a complex system.

But if the system that the forecaster is modelling is complex – by which I mean has unknowable numbers of influences and information, and/or randomness inherent in it – then forecasting models become little better than guesses.

And typically the forecast that models produce in these circumstances often reflect the interests and biases of the forecasters themselves.

Examples of complex systems include: modern economies; stock market and currency prices; as well as the climate. In modelling systems like these, the modeller will never be able to accurately reflect what is going on.

I worked for the Bank of England – economic forecasts cannot be trusted

Models can be amazingly reliable, or little better than a guess

www.telegraph.co.uk

 

[Navy_Pete]

See this in fire all the time; there was a study done a while ago with something like 7 or 8 teams of genuine experts modeling something in a really well denied apartment fire, where they knew exactly where everything was, where the fire started, what the materials were etc.

Essentially they all ended up with different models, which were all wrong in some respects (in different ways). Nothing against them, it's just a very complex system with a lot of interactions that we don't understand and can't model. And that was something that was a relatively simple structural fire.

But the results look impressive in full colour and 3D, so BGHs tend to take them as gospel, while ignoring the uncertainties and real world issues where what is assumed isn't actually true (ie building facing is non-flammable, fire doors are closed, etc). Or change something that was a key assumption later without appreciating that it completely undermines your previous safety case work. It's hard on long term and continuous work to make sure those key assumptions are well understood.

 

[Brad Sallows]

The behaviour of complex systems can inevitably only be mathematically defined by complex systems of differential equations, which means solutions inevitably cannot employ approximations (eg. magical constant parameters and simpler equations) except for small regions of the problem domain. Models are approximations, usually crude ones.

People who haven't studied the mathematics and its associated computer science topics have no grasp of this, nor should one expect them to. If they did, there'd be a lot more common sense skepticism towards models.

 

[Kirkhill]

The behaviour of complex systems can inevitably only be mathematically defined by complex systems of differential equations, which means solutions inevitably cannot employ approximations (eg. magical constant parameters and simpler equations) except for small regions of the problem domain. Models are approximations, usually crude ones.

People who haven't studied the mathematics and its associated computer science topics have no grasp of this, nor should one expect them to. If they did, there'd be a lot more common sense skepticism towards models.

A major problem, IMO, with models is that they are often commissioned by people that do not understand "uncertainty" and wish to create "certainty". Some want certainty to help them make decisions. Others just want to create the impression of certainty to sell decisions.

 

[Colin Parkinson]

For the Port Mann bridge, the proponent hired the best company here to build a model of the river at, up and downstream of the site to determine the effect on the river bed by the placement of piers. The river bed was accurately modelled and the area has been studied heavily over the years. We approved the pier placements based on the modelling to prevent the formations of sandbars downstream. Unfortunately the model was not accurate and sandbars formed where we didn't want them to. Everyone did their best and spent a lot of money on the issue, but the model still failed, despite having a lot of control over the data and inputs. The lesson is that nature is way more complex than we assume. And that our political masters don't want to hear about the complexities and just want simple answers and solutions.

 

[dimsum]

For the Port Mann bridge, the proponent hired the best company here to build a model of the river at, up and downstream of the site to determine the effect on the river bed by the placement of piers. The river bed was accurately modelled and the area has been studied heavily over the years. We approved the pier placements based on the modelling to prevent the formations of sandbars downstream. Unfortunately the model was not accurate and sandbars formed where we didn't want them to. Everyone did their best and spent a lot of money on the issue, but the model still failed, despite having a lot of control over the data and inputs. The lesson is that nature is way more complex than we assume. And that our political masters don't want to hear about the complexities and just want simple answers and solutions.

A lecturer once said “all models are flawed to some extent, but some are useful”.

 

[Kirkhill]

A lecturer once said “all models are flawed to some extent, but some are useful”.

You only know which ones are useful once you have tested out the hypothesis. Economies and Climates present a challenge in that regard. On the other hand, BC will know better when they build the next Port Mann bridge.

 

[Colin Parkinson]

You only know which ones are useful once you have tested out the hypothesis. Economies and Climates present a challenge in that regard. On the other hand, BC will know better when they build the next Port Mann bridge.

No they won't as the corporate knowledge will be gone.

 

[daftandbarmy]

You only know which ones are useful once you have tested out the hypothesis. Economies and Climates present a challenge in that regard. On the other hand, BC will know better when they build the next Port Mann bridge.

“The ideas of economists and political philosophers, both when they are right and when they are wrong are more powerful than is commonly understood. Indeed, the world is ruled by little else. Practical men, who believe themselves to be quite exempt from any intellectual influences, are usually slaves of some defunct economist.”

― John Maynard Keynes

 

[lenaitch]

A major problem, IMO, with models is that they are often commissioned by people that do not understand "uncertainty" and wish to create "certainty". Some want certainty to help them make decisions. Others just want to create the impression of certainty to sell decisions.

Often in areas where certainty cannot, and never will, exist.

When's the last time you heard a politician or bureaucrat say "we think", "based on our best guess", or "there's a good chance"

 

[GR66]

Often in areas where certainty cannot, and never will, exist.

When's the last time you heard a politician or bureaucrat say "we think", "based on our best guess", or "there's a good chance"

During COVID. And how well did that go over for them?