Better Control Groups for In-market Tests

The Opportunity

A retailer was conducting an in-market test of a store format change in a particular designated market area (DMA). Their approach for monitoring the impact was to compare key metrics from these stores against all the remaining, non-test stores in the entire chain.

The metrics were provided to managers through their financial dashboard reporting system, grouped into the test stores and “control” stores, for side-by-side comparison. This included dollar sales and transactions, gross profit, cost of goods, operating expenses, etc.

While the reports showed a large difference between test and control stores, the imbalance between the two store groups cast doubt on the interpretation of the comparisons. They needed a way to be more precise and confident in the results.

Our Approach

We recognized the need for a set of control stores to match the test stores as closely as possible, based on key metrics, such as sales.

We used time series data mining techniques to look every store’s sales pattern over time, and to find each test store’s “nearest neighbor” store, based on the most similar sales pattern. This including looking at patterns in trends, seasonality, sudden shifts, spikiness due to promotions, etc.

With each test store’s nearest neighbor control store identified, a special control group was created for comparisons in the dashboarding tool. This allowed for a better pre-post, test vs. control analysis of the store format change test

The Impact

The original raw comparisons showed improvements, in some cases, of greater than 30%. While managers would have loved to see those kinds of results from a store format change, these results were not believable.

The revised dashboard results, based on a tightly focused, more precisely matched set of control stores, dramatically reduced that impact. But even with much more modest, even small percent differences, the test was considered a success.

In the words of one financial planner, “It’s important that we know the truth rather than inflate expectations with a flawed, raw sales comparison.”