Analyzing A/B test results which are not normally distributed, using independent t-test

Question

I have a set of results from an A/B test (one control group, one feature group) which do not fit a Normal Distribution. In fact the distribution resembles more closely the Landau Distribution.

I believe the independent t-test requires that the samples be at least approximately normally distributed, which discourages me using the t-test as a valid method of significance testing.

But my question is: At what point can one say that the t-test is not a good method of significance testing?

Or put another way, how can one qualify how reliable the p-values of a t-test are, given only the data set?

Answer 1

The distribution of your data doesn't need to be normal, it's the Sampling Distribution that has to be nearly normal. If your sample size is big enough, then the sampling distribution of means from Landau Distribution should to be nearly normal, due to the Central Limit Theorem.

So it means you should be able to safely use t-test with your data.

Example

Let's consider this example: suppose we have a population with Lognormal distribution with mu=0 and sd=0.5 (it looks a bit similar to Landau)

So we sample 30 observations 5000 times from this distribution each time calculating the mean of the sample

And this is what we get

Looks quite normal, doesn't it? If we increase the sample size, it's even more apparent

R code

x = seq(0, 4, 0.05)
y = dlnorm(x, mean=0, sd=0.5)
plot(x, y, type='l', bty='n')


n = 30
m = 1000

set.seed(0)
samp = rep(NA, m)

for (i in 1:m) {
  samp[i] = mean(rlnorm(n, mean=0, sd=0.5))
}

hist(samp, col='orange', probability=T, breaks=25, main='sample size = 30')
x = seq(0.5, 1.5, 0.01)
lines(x, dnorm(x, mean=mean(samp), sd=sd(samp)))


n = 300
samp = rep(NA, m)

for (i in 1:m) {
  samp[i] = mean(rlnorm(n, mean=0, sd=0.5))
}

hist(samp, col='orange', probability=T, breaks=25, main='sample size = 300')
x = seq(1, 1.25, 0.005)
lines(x, dnorm(x, mean=mean(samp), sd=sd(samp)))

Answer 2

Basically an independent t-test or a 2 sample t-test is used to check if the averages of the two samples are significantly different. Or, to put in another words, if there is a significant difference between the means of the two samples.

Now, the means of those 2 samples are two statistics, which according with CLT, have a normal distribution, if provided enough samples. Note that CLT works no matter of the distribution from which the mean statistic is built.

Normally one can use a z-test, but if the variances are estimated from the sample (because it is unknown), some additional uncertainty is introduced, which is incorporated in t distribution. That's why 2-sample t-test applies here.