Approaches to A/B testing when you can't randomize on the user level

Question

I am trying to find suggestions for different approaches to running tests when it isn't possible to randomly assign users to test and control buckets.

For example, if I own 10 shops and have a certain algorithm that I use to price items, and perhaps I want to test a new algorithm that would result in price changes. There are a few approaches I can see but it isn't clear to me if I'm missing others and would love to hear more.

Standard A/B testing:

If I were to randomly assign shoppers to test or control buckets then this would obviously be a bad experience for them (given that prices are visible).

Switchback experiment

Perhaps instead I decide to run a switchback experiment, in which the control pricing algorithm runs for 6 hours, then the test pricing algorithm, runs for 6 hours, and so on throughout my test period. This is obviously better than the standard A/B approach but I feel like it still has problems (ie what if the price change results in a change to long term customer behavior but it takes some time for the behavior to change - perhaps the switchback wouldn't capture this).

Causal Impact approach

Another approach might be to use a Causal Impact style analysis, in which I set certain shops as 'control' shops, and other shops as 'test' shops, and then change the algorithm at the test shops and analyze my metric of interest, where we look for a causal effect in the time series at the test shops with respect to the control shops. The drawback here is that you seem highly dependent on how well the controls represent the test shops.

Does anyone have any other suggestions, or even somewhere I can read more about different experimental setups, beyond A/B testing?

Answer 1

I would recommend against switch back as the purchase data is likely to exhibit seasonality (e.g. 2pm - 6pm / tuesdays exhibit lower volume, December exhibit higher volumes, company running advertisements during certain months etc.), which would become an unnecessary exogenous factor that complicates the post implementation analysis.

I am inclined towards the causal impact approach. However, there are certain things that should be checked before selecting the stores as control vs. test (it is a kick-start list, please feel free to adapt to your business domain).

1. Decide on the KPI to be measured as the objective, is that revenue per customer? Profit per customer? Or total profit per store? Is the algorithm designed to increase per customer KPI or more customers?

2. Check that the stores selected have some history to give assurance that they were similar in terms of the selected KPI before the implementation of the tests. Remember to remove outlier customers / transactions.

3. Check that enough stores are selected in control vs. test buckets to detect a desired threshold increase that is statistically significant.

Hope this helps! :)