Scenario Forecasting Using Bayesian Modelling
By Jakob Nyström, Dan Sack, Felix Martinsson, and Aaron Arnoldsen
Over the past years, machine learning (ML) has transformed the ability of companies to forecast demand. Instead of relying solely on extrapolation using a few drivers and “gut feel,” business leaders can now use an unprecedented amount of data to create sophisticated statistical models. In combination with advanced optimization algorithms, leaders can now make more effective decisions. The use cases are many, including end-to-end supply chain planning, allocation of new products, and pricing and promo decisions.
However, the Covid-19 pandemic has highlighted the fact that traditional ML models struggle to account for unexpected, rapid changes in consumer behavior and demand. Looking at the retail sector, Covid has caused a massive drop in demand for fashion and luxury goods. In contrast, and despite store closures, demand for home decoration and DIY has surged. Secondary infection waves and local outbreaks, combined with restrictions and lockdowns of varying degrees, have made demand extremely hard to predict.
But this is not just a Covid issue. What the pandemic has done in the past year to corporate forecasting and planning is just one example (granted, an extreme one) of how large shocks and shifts in demand can cause unanticipated impacts to highly complex, interconnected systems and supply chains.
The Way Forward: Novel Data Sources and Structurally Different Models
Where should businesses and other organizations turn when it seems traditional models are rendered obsolete? The answer is two-fold:
1. Leverage new data sources that are forward-looking, local, and close to real-time to enhance existing ML models and enable more accurate scenario planning.
2. Explore the use of Bayesian hierarchical models to make better predictions. These models incorporate human knowledge into their structures, can work with more sparse data, and are probabilistic in their estimations.
To address the first point, BCG has during 2020 developed the Lighthouse by BCG forecasting engine and framework. Lighthouse improves demand prediction by using new sets of close-to-real-time (high-frequency) data sources such as consumer mobility trends, economic activity tracking, web traffic and search trends, and government and publicly available spending and unemployment figures.
Given the disruptions caused by the pandemic, it may be tempting to simply exclude 2020 from training data and conduct top-down adjustments. But this relies on a strong assumption that things will return to normal. Instead, by combining external and internal data to understand how different indicators impact demand, the Lighthouse engine is able to predict shifts in demand more accurately and rapidly than ML models that rely on large sets of historical training data alone.
This enables it to both produce short-term demand predictions and help with medium- and long-term scenario planning. This approach was successfully deployed at more than 100 BCG clients in 2020. In the example below, the use of Lighthouse resulted in a 50% reduction in prediction error.
Recent advances in computing power also make it possible to address the second point: taking advantage of the properties of hierarchical Bayesian models, which are arguably among the more important statistical advances in the past 50 years. The use of previous knowledge and human intuition via so-called “priors” in such models delivers two principal benefits:
1. First, we can embed human experience and knowledge into the structure of the models (where and when we trust it).
2. Second, we can make accurate predictions with far fewer data points than are required by traditional ML models, allowing us to both improve existing data and analytics products and tackle historically untouched business problems.
This approach does not mean that traditional ML models relying on historical data are obsolete. For example, insights gained from historical sales may help us understand how certain product features drive demand, which is still useful when forecasting for new products. But companies that continue to rely only on historically-oriented approaches such as this risk falling behind, both during the current disruption — and when the next one around the corner arrives.
The Bayesian Approach: More Power to the People — and to the Data
In most approaches to analytical modeling, the model itself is relatively “un-opinionated” and naïve to the structure of a problem. In many cases, that is a good thing: If sufficient data is available, such as in the case of computer vision or a company in a very stable market environment, the structure can often be inferred without significant bias.
However, when data is limited or of questionable quality or relevance (such as in a post-Covid world in which historical data may not represent future behaviors, trends, or patterns), it can be advantageous to directly embed structure and beliefs into the model.
Bayesian modeling returns humans to the center of the modeling process in fundamental ways:
a) In a Bayesian model, the business user and data scientist define their own prior beliefs before training the model.
b) These prior beliefs may encompass specific business understanding; more broadly a set of structures in how variables relate to one another.
c) After specifying the prior, the data is incorporated during training and the model will give an informed estimation by leveraging both priors and data.
The power of this approach is that the model must no longer infer all the structure by relying purely on what have traditionally been massive amounts of historical data. Instead, the model combines the intuition and experience of those who developed and provided input into the model with available data. This approach enhances the power of these models to address increasingly common situations in which data is limited, of low quality, or both. By defining a clear structure in the data, it is possible to estimate many variables even when data is sparse.
The approach also provides a probabilistic estimation of uncertainty. First, it provides a much better understanding of the confidence in the predictions. And when the business decision is asymmetric (such as when understocking may be more costly than overstocking), we can easily adjust our decision based on the probability distribution.
Bayesian models also help overcome a common hurdle to business acceptance of AI — the feeling that algorithms are hard-to-understand “black boxes.” And, through their probabilistic estimations, these models provide a solid foundation for scenario planning.
The Science of Scenarios
The entire purpose of creating business scenarios is to account for uncertainty. Traditionally, scenarios are created simply by subjectively picking a few select drivers and varying assumptions to get base, low, and high cases. Often, in today’s reality, that is not enough.
By taking a Bayesian approach, scenario planning has potential to become more science than art:
a) Uncertainty about underlying drivers going forward can be embedded into the model through different sets of priors (these can be considered as scenario assumptions)
b) The output of a Bayesian model is a probability distribution, not just a point estimate. In combination with the use of multiple priors, this means that we get a number of scenarios as output. These range from the most likely base case to more- or less-likely deviations from it.
c) Additionally, using the estimates from that probability function in an optimization algorithm offers an alternative to doing computationally-costly stochastic optimization.
Bayesian modeling can also be combined with better use of local and high-frequency data from the Lighthouse suite. For example, if we have a probabilistic scenario estimation of Covid-19 cases and a demand forecasting model that ties demand to the number of cases on a city level, we can get the best estimate of total demand. We can then use the estimate to, for example, optimize the replenishment of stores in different cities within a market. As always, the real value of demand prediction comes when using forecasting and scenario generation to improve decision making.
Pandemics are not the only use case for Bayesian models. For example, we recently developed a Bayesian hierarchical time-series model for a large retailer that estimates the effects of store openings in a market. The model creates a full omnichannel ecosystem view that predicts both new sales and cannibalization of existing channels to get the net effect on sales and profitability — including a view of uncertainty.
The issue of predicting new sales and channel cannibalization is a business problem for which data can be sparse, especially when there have been only a limited number of store openings of a given format. In this case, country-level estimates can be used as priors to estimate effects at the city level — something that would not be a straightforward process when using traditional ML models. Similarly, results from one market can be used as priors when expanding to other markets. For example, we could embed a strong belief that the distance between stores and residential areas versus the distance between stores themselves has roughly the same effect on demand across markets.
Preparing for Future Uncertainty
Regardless of the specific cause of disruption, companies able to de-average demand — and quickly identify where and how demand is changing — are winning the battle, and will continue to do so long after this pandemic has passed. In a world that is becoming increasingly complex and economically and politically uncertain, adopting new approaches to forecasting and scenario planning is a key lever that organizations can employ to improve their decision making.
The answer to the challenge of forecasting amid instability is bionic: More sophisticated forecasting algorithms that leverage a larger set of granular and close-to-real-time data sources, and a higher degree of human intuition and knowledge in their structure. By taking such an approach, companies can regain or expand their competitive advantage.
