I am trying to find suggestions for different approaches to running tests when it isn't possible to randomly assign users to test and control buckets.
For example, if I own 10 shops and have a certain algorithm that I use to price items, and perhaps I want to test a new algorithm that would result in price changes. There are a few approaches I can see but it isn't clear to me if I'm missing others and would love to hear more.
Standard A/B testing:
If I were to randomly assign shoppers to test or control buckets then this would obviously be a bad experience for them (given that prices are visible).
Perhaps instead I decide to run a switchback experiment, in which the control pricing algorithm runs for 6 hours, then the test pricing algorithm, runs for 6 hours, and so on throughout my test period. This is obviously better than the standard A/B approach but I feel like it still has problems (ie what if the price change results in a change to long term customer behavior but it takes some time for the behavior to change - perhaps the switchback wouldn't capture this).
Causal Impact approach
Another approach might be to use a Causal Impact style analysis, in which I set certain shops as 'control' shops, and other shops as 'test' shops, and then change the algorithm at the test shops and analyze my metric of interest, where we look for a causal effect in the time series at the test shops with respect to the control shops. The drawback here is that you seem highly dependent on how well the controls represent the test shops.
Does anyone have any other suggestions, or even somewhere I can read more about different experimental setups, beyond A/B testing?