Questions tagged [ab-test]

A/B testing, also known as split or bucket testing, is a controlled comparison of the effectiveness of variants of a website, email, or other commercial product.

Filter by
Sorted by
Tagged with
1
vote
0answers
12 views

How to create A/B test segements for highly variable data

I have a data in which there is a high degree of variability. My Objective is to do an AB test to check the behavior change due to new changes. All samples has shown historically high and low ...
0
votes
2answers
35 views

A/B testing with non-Gaussian distributions

I have two sets of samples (A, B) with a relatively high number (~10,000) and I want to see if a factor has affected sample B or not. Naturally, I should use A/B testing. The problem is, the ...
0
votes
0answers
14 views

Help in interpreting the results of the A / B test Bayessian approach for non binomial data

I would like to carry out an A / B test based on data regarding the profit from a given user. I found an interesting library for such tests https://github.com/tcassou/babtest. Now I would like to ask ...
1
vote
0answers
12 views

Is it necessary to cap test and control metric at same percentile to treat outliers?

I am working on an AB test and the metric I am using to compare test and control is revenue. Both of my test and control have outliers. I am trying to treat outliers and have found that to do so, I ...
1
vote
0answers
21 views

How to design an A/B test in practice

I am a statistician who understands hypothesis testing in general, but is completely new to A/B testing in a commercial context. Could someone explain in any detail, how to design a test for a ...
2
votes
1answer
54 views

How to select random data for two different recommender systems?

The business problem: We have two different vendors that offer personalized recommender engines and want to do A/B testing with them. The recommendation will give the user a personalized offer via a ...
1
vote
0answers
24 views

Significant testing - repeated observation over multiple days

I work in mobile gaming, and want to analyze A/B test groups, but I believe I'm introducing errors in my calculations. The metric I'm looking at is: num of unique players who engaged in battle that ...
1
vote
0answers
112 views

How to verify A/B test

Is there a way to check the result of A/B test to determine the better one. Suppose, Group A has 100 views and 10 clicks. Group B has 10 views and 2 clicks. How to chose which one produces more ...
-1
votes
1answer
44 views

Testing whether observed control/test split invalidates my assumption of 50/50 randomised traffic

I recently ran a test on site traffic that was designed to be a 50/50 split. I actually observed a 50.7% to 49.3% split (154,490 vs 150,151 visitors). How can I test based on the sample size whether ...
4
votes
1answer
250 views

A/B testing: How to calculate p-value on post test segments?

My question on A/B testing is about doing post test segmentation analysis. For example: I run an A/B test on my website to track bounce rate. On the treatment group, i put a video to explain ...
1
vote
1answer
96 views

A/B Testing Suitability for Hypothesis

I'm obscuring the data I'm working on, but I think this should get the point across. I'm trying to design a test for the following: A client sells various popsicles. They think that on days when it ...
3
votes
2answers
117 views

Is it scientifically correct to derive conclusions unrelated to hypothesis from A/B test data

Consider a software A/B test with the hypothesis that "the addition of feature F is predicted to increase metric X". At the end ...
1
vote
0answers
53 views

How can one compare these groups?

I have two dataset A and B. We have 93 persons in A and 88 in B. Dataset A contains number of person who had made a payment. So A is ...
1
vote
3answers
668 views

How to approach a Data Science case study question?

I recently had a phone interview with a consumer tech company for a quant position. The question was basically, "imagine a facebook style social network site. Six months ago a new feature called '...
0
votes
1answer
696 views

Approaches to A/B testing when you can't randomize on the user level

I am trying to find suggestions for different approaches to running tests when it isn't possible to randomly assign users to test and control buckets. For example, if I own 10 shops and have a ...
5
votes
1answer
2k views

What does Prec@1 in fastText mean?

In Bag of Tricks for Efficient Text Classification paper which is popular right now, he calculates prec@1 for the datasets in the experimentation segment. What does that mean?
5
votes
2answers
485 views

AB testing : When AA testing doesn't work

After 6 months of AB testing on our CRM tool (Oracle Responsys, but this could be true with anyone), the test exhibited some weird results so we decided to pause everything, and to make some good old ...
0
votes
1answer
57 views

Advance Methods of Understanding Significance of Customer Behaviors

I currently own a couple of websites and lately I've been implementing some feature changes - I've noticed some changes in website traffic and I was wondering what some of the more sophisticated ways ...
1
vote
0answers
187 views

A/B Test: Determine if a Lift in Measured Sales is Significant

I am no stats expert so any assistance will be greatly appreciated. I’m working on a post-campaign analysis to determine if any measured sales lift (sales lift per audience) is significant between ...
4
votes
1answer
823 views

A/B testing randomization step

Let's say I want to measure a medicine’s impact on height. So I randomly broke down my user base into two groups (control and experiment, obviously). I calculated the average height for both groups ...
2
votes
4answers
3k views

What is the minimum size of the test set?

The mean of a population of binary values can be sampled with about 1000 samples at 95% confidence, and 3000 samples at 99% confidence. Assuming a binary classification problem, why is the 80/20% ...
3
votes
1answer
2k views

How would I chi-squared test these simple results from A/B experiment?

I have results from an A/B experiment where users could do one of three things: Watch, Interact, or Nothing My data is like this: ...
2
votes
2answers
260 views

Retail Store Testing

I am an analyst who is new to retail store testing (for example: customer response to a store new layout or a new product or process in one or 2 stores). Any suggestions for a good book/blog/any ...
1
vote
0answers
88 views

Campaign Hypothesis Testing: Is using chi-square test appropriate?

I’ve just gotten the results back from a marketing campaign that consisted of testing new men’s and women’s apparel creative (test creative) against business as usual men’s and women’s creative (...
1
vote
2answers
198 views

How to determine if a company decision was successful or not?

I'm trying to figure out if a decision taken in a company (offering discounts for specific products) is successful or not. I have done some research and saw that A/B testing might be a way to do this ...
4
votes
3answers
2k views

What are the methods to ensure that the population split for A/B test is random?

Before launching an A/B test, what are the methods to ensure that the population split in control and target group is random for a particular label say, purchase rate.
2
votes
2answers
235 views

How to control false positives in sequential A/B testing while keeping a low sample size?

I am working on a planned sequence of n independent A/B tests(=run a maximum of n tests or stop earlier if a good improvement is found) and in order to keep the significance level within an acceptable ...
2
votes
3answers
2k views

Analysis of Split (A/B) tests using Poisson and/or Binomial Distribution

Cross posting this from Cross Validated: I've seen this question asked before, but I have yet to come across a definitive source answering the specific questions: What's the most appropriate ...
13
votes
2answers
2k views

Analyzing A/B test results which are not normally distributed, using independent t-test

I have a set of results from an A/B test (one control group, one feature group) which do not fit a Normal Distribution. In fact the distribution resembles more closely the Landau Distribution. I ...