Questions tagged [ab-test]
A/B testing, also known as split or bucket testing, is a controlled comparison of the effectiveness of variants of a website, email, or other commercial product.
55
questions
0
votes
1
answer
21
views
Evaluate a Recommender System based on the data between two months
currently my company's planning to use a new Recommender tool/library for a book website, and now we want to compare the result between these two tools (both of the tool use Universal Recommender ...
0
votes
0
answers
13
views
A/B test results don't hold in hold-out dataset
I'm working on a project consisting in analyzing some game data. The game consists of two players competing between them. The result of the game is either win or lose. Players can choose from ...
- 536
0
votes
1
answer
20
views
Why NHST based A/B testing is not allowed to continuously monitor?
I am reading a paper regarding A/B test continuous monitoring. https://alexdeng.github.io/public/files/continuousMonitoring.pdf
In the abstract part, it states
A crucial problem of
Null Hypothesis ...
- 101
0
votes
0
answers
29
views
A/B testing for channel preference
Right now my organization runs a lot of promotional campaign every month. We send email, SMS and WhatsApp to all customers for each campaign. I am running a project to identity the best channel for a ...
0
votes
2
answers
81
views
How to check the impact of predictive Model
Suppose that I build an email marketing campaign where the goal is to predict which members will make purchases or not and based on that change the marketing email campaigns. After we build the model, ...
- 11
0
votes
0
answers
17
views
Interview Question: A/B Survey Results Non-Random Sampling
How would you respond to the following case/question:
Assume you are testing the effect of an ad campaign at a social media company. The study is divided into two groups, 5,000 subjects who are shown ...
0
votes
0
answers
10
views
A/B Testing Disruptive Changes/New Features
So I work for a company which is building a product.
Soon, we'll release an "all-new UI" which is supposed to be ground-breaking.
We want to measure the impact of introducing this new UI by ...
- 1
0
votes
0
answers
3
views
How to factor in pre test differences among variants of a multi variant test in its analysis?
I have data from a Control/A/B/C test. This test was run for 30 days with a 10/30/30/30 split of audience where each variant was given an offer of different price with same discount. Before conducting ...
0
votes
0
answers
20
views
Hypothesis test for classification model
I have a model that outputs 0 or 1 for interest/not-interest in a job. I'm doing an A/B/C test comparing two models (treatment groups) and none (control group).
ANOVA for hypothesis testing and t-test ...
- 725
1
vote
0
answers
7
views
Can I use multi armed bandits to optimize how much both algorithms are weighted when creating a composite score?
So, I'm aware that multi-armed bandits are great for evaluating multiple models and from what I understand, it is mainly used to pick a specific model.
I would still like to evaluate two models but I ...
- 11
0
votes
1
answer
41
views
A/B test on model - split on results
I developed a predictive model that assigns the best product (P1, P2, P3) for each customer. I wanted to compare the conversion rate using this model VS the as-is deterministic assignment, so I ...
- 123
1
vote
0
answers
28
views
Causal Inference where the treatment assignment is randomized [closed]
I have mostly worked with Observational data where the treatment assignment was not randomized. In the past, I have used PSM, IPTW to balance and then calculate ATE.
My problem is:
Now I am working on ...
- 111
2
votes
1
answer
789
views
AB testing split algorithm
I want to understand what is the most effective algorithm for splitting. I have ids of users and I want to split them into 2 groups.
Now I have 2 variants:
Modulo approach - let's say we will place ...
- 123
0
votes
0
answers
134
views
Multivariate testing
I'm going to run a test with 4 different variants (3 variants and a control group), and we want to find the variant with the highest conversion.
Are there any resources/methods in R/python to:
...
- 5,921
0
votes
1
answer
51
views
How do I conduct an experiment on the new pricing if it's impossible to conduct an A/B test?
We want to introduce a new price list for the customers of our international SaaS company.
Beforehand we want to test this new price list in several countries.
A/B test cannot be conducted here ...
1
vote
0
answers
64
views
Practical constraints in A/B testing
I saw an article about an A/B test that google had performed way back. They wanted to decide what shade of blue a button should be and how that affects click-through rate. They divided users randomly ...
- 157
0
votes
0
answers
50
views
What is the right approach to bucket users for algorithms with different coverage for A/B testing
I've couple of recommendation algorithms that I want to A/B test. Algorithm A has 90% user coverage and algorithm B has 95% user coverage. That means if the algorithms are asked to provide ...
- 571
2
votes
2
answers
750
views
A/B Testing (Binomial Distribution vs Random Distribution)
When performing an A/B test for the number of clicks for users viewing (each view is an impression) two variants of an ad, a binomial distribution can be assumed where each variant has a constant ...
- 121
2
votes
1
answer
116
views
A/B test results contradictory with offline machine learning model performance
This seems to be a common problem when bringing machine learning models to production.
Let's say we have an optimized machine learning model which gives decent performance metric in the unseen testing ...
- 251
1
vote
1
answer
52
views
Time duration for ML models A/B testing
I am going to perform A/B tests for ML models. However I am not sure how long should I run it online in order to see significant differnce. What would be the right time frame ? and what will be the ...
- 153
0
votes
1
answer
25
views
Recommend System AB test metric events
I build personal recomendation system for choosing games. In website on main page on special place there is collection of personal games recomendation. And after AB test(between 2 recommend system) I ...
- 1
1
vote
0
answers
22
views
Data science Product Question Help [closed]
I was recently asked this in a ds interview pertaining to product thinking -
If a product feature is being rolled out, but an a/b test is unable to be performed due to whatever reason, how can we ...
- 11
0
votes
0
answers
30
views
Imbalanced classification or Regression? What is the best approach to my A/B testing related problem?
The context of the problem is A/B testing of two new versions of a game. I have a structured dataset (50000 rows x 22 columns) from the game designers that represents data with respect to two versions ...
2
votes
1
answer
95
views
Permutation test on two groups
I am trying to use a permutation test to test my hypothesis.
I want to make sure I am understanding concept of permutation correctly.
I have control and experimental group. Then I combine them and ...
- 385
1
vote
1
answer
144
views
Treatment and Control selection in A/B Testing
I'm hoping to get a better understanding of A/B Testing design. In particular, I'm interested in understanding how treatment and control units are selected. I read that these 2 groups are selected ...
- 172
1
vote
0
answers
162
views
Firebase AB testing algorithm
We have run an AB test at firebase which has the following results:
I was also building my own Bayesian AB-test suite and was wondering how they came to these conclusions.
What I was doing was ...
- 73
1
vote
1
answer
127
views
No statistical significance but observable trends
I have a general inference question regarding scenarios when results from data are not statistically significant but there appears to be an observable trend.
For example, ...
- 133
1
vote
2
answers
41
views
Fitting a model as a significance testing alternative
My colleagues use somewhat unusual approach to estimate how far away performance of some of our company related processes diverge from historical levels. Their nature isn't really relevant for the ...
- 113
2
votes
1
answer
26
views
How to create A/B test segements for highly variable data
I have a data in which there is a high degree of variability.
My Objective is to do an AB test to check the behavior change due to new changes.
All samples has shown historically high and low ...
- 21
1
vote
3
answers
178
views
A/B testing with non-Gaussian distributions
I have two sets of samples (A, B) with a relatively high number (~10,000) and I want to see if a factor has affected sample B or not. Naturally, I should use A/B testing. The problem is, the ...
- 111
1
vote
0
answers
26
views
Is it necessary to cap test and control metric at same percentile to treat outliers?
I am working on an AB test and the metric I am using to compare test and control is revenue. Both of my test and control have outliers. I am trying to treat outliers and have found that to do so, I ...
- 11
2
votes
1
answer
66
views
How to select random data for two different recommender systems?
The business problem: We have two different vendors that offer personalized recommender engines and want to do A/B testing with them. The recommendation will give the user a personalized offer via a ...
1
vote
1
answer
57
views
Significant testing - repeated observation over multiple days
I work in mobile gaming, and want to analyze A/B test groups, but I believe I'm introducing errors in my calculations.
The metric I'm looking at is:
num of unique players who engaged in battle that ...
- 11
2
votes
1
answer
359
views
How to verify A/B test
Is there a way to check the result of A/B test to determine the better one.
Suppose,
Group A has 100 views and 10 clicks.
Group B has 10 views and 2 clicks.
How to chose which one produces more ...
- 687
-1
votes
1
answer
108
views
Testing whether observed control/test split invalidates my assumption of 50/50 randomised traffic
I recently ran a test on site traffic that was designed to be a 50/50 split. I actually observed a 50.7% to 49.3% split (154,490 vs 150,151 visitors).
How can I test based on the sample size whether ...
8
votes
2
answers
1k
views
A/B testing: How to calculate p-value on post test segments?
My question on A/B testing is about doing post test segmentation analysis.
For example:
I run an A/B test on my website to track bounce rate. On the
treatment group, i put a video to explain ...
- 233
1
vote
1
answer
160
views
A/B Testing Suitability for Hypothesis
I'm obscuring the data I'm working on, but I think this should get the point across. I'm trying to design a test for the following:
A client sells various popsicles. They think that on days when it ...
- 2,051
3
votes
2
answers
124
views
Is it scientifically correct to derive conclusions unrelated to hypothesis from A/B test data
Consider a software A/B test with the hypothesis that "the addition of feature F is predicted to increase metric X".
At the end ...
- 133
1
vote
0
answers
54
views
How can one compare these groups?
I have two dataset A and B. We have 93 persons in A and 88 in B.
Dataset A contains number of person who had made a payment. So A is
...
- 197
2
votes
3
answers
947
views
How to approach a Data Science case study question?
I recently had a phone interview with a consumer tech company for a quant position. The question was basically, "imagine a facebook style social network site. Six months ago a new feature called '...
- 521
1
vote
1
answer
2k
views
Approaches to A/B testing when you can't randomize on the user level
I am trying to find suggestions for different approaches to running tests when it isn't possible to randomly assign users to test and control buckets.
For example, if I own 10 shops and have a ...
- 1,833
7
votes
1
answer
4k
views
What does Prec@1 in fastText mean?
In Bag of Tricks for Efficient Text Classification paper which is popular right now, he calculates prec@1 for the datasets in the experimentation segment. What does that mean?
- 2,296
5
votes
2
answers
711
views
AB testing : When AA testing doesn't work
After 6 months of AB testing on our CRM tool (Oracle Responsys, but this could be true with anyone), the test exhibited some weird results so we decided to pause everything, and to make some good old ...
- 191
0
votes
1
answer
71
views
Advance Methods of Understanding Significance of Customer Behaviors
I currently own a couple of websites and lately I've been implementing some feature changes - I've noticed some changes in website traffic and I was wondering what some of the more sophisticated ways ...
- 143
1
vote
0
answers
261
views
A/B Test: Determine if a Lift in Measured Sales is Significant
I am no stats expert so any assistance will be greatly appreciated.
I’m working on a post-campaign analysis to determine if any measured sales lift (sales lift per audience) is significant between ...
- 63
4
votes
1
answer
1k
views
A/B testing randomization step
Let's say I want to measure a medicine’s impact on height. So I randomly broke down my user base into two groups (control and experiment, obviously). I calculated the average height for both groups ...
- 143
5
votes
4
answers
9k
views
What is the minimum size of the test set?
The mean of a population of binary values can be sampled with about 1000 samples at 95% confidence, and 3000 samples at 99% confidence.
Assuming a binary classification problem, why is the 80/20% ...
- 65
3
votes
1
answer
3k
views
How would I chi-squared test these simple results from A/B experiment?
I have results from an A/B experiment where users could do one of three things: Watch, Interact, or Nothing
My data is like this:
...
- 133
2
votes
2
answers
334
views
Retail Store Testing
I am an analyst who is new to retail store testing (for example: customer response to a store new layout or a new product or process in one or 2 stores).
Any suggestions for a good book/blog/any ...
- 131
1
vote
0
answers
96
views
Campaign Hypothesis Testing: Is using chi-square test appropriate?
I’ve just gotten the results back from a marketing campaign that consisted of testing new men’s and women’s apparel creative (test creative) against business as usual men’s and women’s creative (...
- 63