How do I conduct an experiment on the new pricing if it's impossible to conduct an A/B test?

Question

• We want to introduce a new price list for the customers of our international SaaS company.
• Beforehand we want to test this new price list in several countries.
• A/B test cannot be conducted here because it is forbidden to have different prices for different customers of the same country.
• Thus we want to introduce the new pricing policy in several countries and then figure out whether the new one is better than the old one.

My questions are:

• How to make a conclusion that the new pricing list is better than the old one?
• Should we compare new metrics to the old metrics from the same country? Or should we compare the metrics to the metrics of other countries for the same period of time?
• Which keywords can I use to find detailed information considering this topic? Do some specific statistical tests exist for such non-trivial cases?

Answer 1

Given the restriction that there can not be different prices for different customers of the same country. There are two primary options:

1. Sequential testing - For each country, all customers get the same price but the price systematically varies over time. This is often called pre and post testing.

2. Cluster testing - Partition countries into two relatively similar groups. One set of countries is "A" and a different set of countries "B". Each "A" customer gets the same price, and each "B" customer gets the same price. However, "A" and "B" countries get different prices. This is often called matched pairs design.

Sequential testing will have the most transfer since it directly evaluations the price in a country. However, it takes longer and there might time-based effects. Often there are other factors that prevent sequential testing (e.g., if customers know the prices will change, they might wait).

Cluster testing works as well as the similarity between the two groups. Analysis is the same as standard A/B testing (assuming you do want to estimate the effect of country).