Augmenting Statistical Forecasting with Machine Learning

The Opportunity

This company has a very capable data science team which looks for ways to bring machine learning into business processes. Until recently, techniques such as deep learning neural networks had not shown success in times series forecasting problems.

We too had always been skeptical of claims of such methods applied in a forecasting context but became aware of new approaches that were starting to show promise.

We invited the company to participate in our Summer R&D program, where we employ graduate student researchers in cooperation with North Carolina State University to investigate topics such as these.

Our Approach

The company already had a sophisticated statistical forecasting system that we had built years before. It had excellent forecast accuracy, but there were important, high-volume SKUs that consistently proved difficult to forecast via statistical models. We focused on thirteen SKUs that were of particular concern among demand planners.

Our goal was not to supplant the statistical forecasting system, but to augment it in situations where traditional methods were falling short.

With historical data on the thirteen SKUs across ten key customers, we built a machine learning pipeline, based primarily on a type of recurrent neural network known as long short-term memory (LSTM).

The Impact

On a four-week holdout period of weekly data, we found that the LSTM approach could provide better forecasts than the statistical system 64% of the time. Note that the statistical system was not a single model, but a sophisticated “pick best” system that evaluates all common types of times series forecasting models.

We assessed the accuracy impact when the LSTMs did better. As these were difficult SKUs to forecast, the average MAPE (Mean Absolute Percent Error) from the statistical system was high: 97.2. The LSTM system achieved 47.3, cutting the error nearly in half. This translates to supply chain savings of millions of dollars for these SKUs.

Even when the LSTMs did not beat the statistical system, the gap was not as wide: 26.4 for the statistical system versus 35.7 for LSTMs.