1
$\begingroup$

I'm trying to figure out if a decision taken in a company (offering discounts for specific products) is successful or not. I have done some research and saw that A/B testing might be a way to do this but A/B tests can only be carried out having 2 groups (control and experiment), and in this case all I have is past and "after the decision" data.

Can the statistics behind AB testing be used in this case?

The data I have is in the for of sales per day per article only for one company before and after the decision (only one decision).

Example:

sales for articles X and Y from the 2016-01-01 to 2016-02-01. The decision was to lower discounts on articles A and B on 2016-01-15, so I want to know if sales decreased or not after this decision.

$\endgroup$
3
  • $\begingroup$ What does your data look like? Sales per day for a number of days before and after the decision event, and the decision event day, all repeated for a number of companies and some number (>=1) decision events per company? $\endgroup$
    – Spacedman
    Mar 21 '16 at 8:45
  • $\begingroup$ yes, sales per day per article only for one company before and after the decision (only one decision). I edited it also in the question $\endgroup$
    – johan855
    Mar 21 '16 at 9:00
  • 1
    $\begingroup$ Any reason you can't just use a t-test? $\endgroup$
    – Spacedman
    Mar 21 '16 at 9:09
3
$\begingroup$

You could use time series approach to model the "what if not" scenario and compare it with your values after the new program introduction. Check the causal_impact packet from Google and you might also find helpful this tutorial for probabilistic programming. http://nbviewer.jupyter.org/github/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/blob/master/Prologue/Prologue.ipynb

$\endgroup$
5
  • 2
    $\begingroup$ Why not use statistics! Assume the sales before and after the event have a different mean and possibly different variance, then you can test the hypothesis of different means with a Student's t-test. Basic stats 101. Now, if those assumptions aren't valid, we should find out why. $\endgroup$
    – Spacedman
    Mar 21 '16 at 14:32
  • $\begingroup$ The mean and variance are in general changing with the time. The OP needs to isolate the effect of his program from this development. $\endgroup$
    – Diego
    Mar 21 '16 at 22:43
  • $\begingroup$ where's your evidence for that? The OP has provided no data or plots. And given that there's only 15 data points before and 15 after the decision point, a fairly simple statistical model should do the trick. Seeing the data would be useful. $\endgroup$
    – Spacedman
    Mar 22 '16 at 8:07
  • $\begingroup$ I speak from my experience. The problem with mean before vs mean after is that it discards the time component. Even if there is a difference it might not be due to the treatment but to a trend or seasonality. And trends and seasonality is what rules sales, again in_general . If you can offer a better i.e. more specific answer what prevents you from doing so? $\endgroup$
    – Diego
    Mar 22 '16 at 13:08
  • 1
    $\begingroup$ I asked the OP "any reason you can't just use a t-test?" and received no clarification. Anyone who proposes an analysis method without so much as a sniff of a look at the data is likely to be barking up something that isn't even a tree. $\endgroup$
    – Spacedman
    Mar 22 '16 at 15:41
1
$\begingroup$

You are trying to determine if some event/treatment (i.e. going on sale) has a significant effect on sales of some product_x. An A/B test is appropriate to answer this question if you have both test and control groups.

Given: (T)est and (C)ontrol groups, where O1 = observation 1, a baseline measurement of normalized sales (i.e. average sales of product_x over some pre-defined time period) and O2 = after treatment measure of normalized sales. Between the first and second observation/measurement, T are given experimental treatment X.

Typical experimental design -

T: O1 X O2

C: O1 . O2

From the OP's description it sounds as if you don't have an adequate control group - although not ideal, it is possible to create control-like groups after the fact (sometimes you can use packages to help create matching control groups after the fact e.g., {optmatch}). In your case, is it possible to get sales data from a similar store to the one you are studying, but one which did not have the sale on product_x? If so, this may be adequate as a control group.

From your description, however, it sounds like you have a pre-test post-design:

Given: (T)est group with treatment X and normalized sales before and after the event

-----time----->

T: O1 X O2

With this design, the primary threat to validity is lack of an adequate control group to ensure that any observed effect is actually due to the experimental treatment (i.e. the Sale). For example, if you observe an increase in over all sales of product_x from the pre to the post-sale period, this finding is consistent with your original hypothesis (A Sale will increase overall sales of product_x), but it is also consistent with some sort of temporal effect that happens to correlate with sales (e.g. seasonality effects, where product sales increase during certain time periods like the holiday season).

$\endgroup$
1
  • $\begingroup$ As I understand OP doesn't have a Control group, that's why I suggest he builds the baseline with a timeseries approach. $\endgroup$
    – Diego
    Mar 29 '16 at 11:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.