Finding P value - Explain

Question

def get_pvalue(con_conv, test_conv,con_size,  test_size,):  
    lift =  - abs(test_conv - con_conv)
    scale_one = con_conv * (1 - con_conv) * (1 / con_size)
    scale_two = test_conv * (1 - test_conv) * (1 / test_size)
    scale_val = (scale_one + scale_two)**0.5
    p_value = 2 * stats.norm.cdf(lift, loc = 0, scale = scale_val )
    return p_value

I have this function and I would like to know what it is actually doing and how it is actually calculating the p-value.

This is to find the difference between the conversion rate of control and test and group from an A/B test.

con_conv --> Conversion rate for control group
test_conv --> Conversion rate for test group
con_size --> population size for control group
test_size --> population size for test group

I understand that scale_one and scale_two are calculating the variance for each group, but I don't understand why they are adding both of them to calculate the standard deviation and why they are multiplying the cdf with 2 to get the p_value.

Answer 1

p_value = 2 * stats.norm.cdf(lift, loc = 0, scale = scale_val )

This is the key for your question: The p-value is the probability that the null hypothesis is true.

If the null hypothesis is true: Your model does not find any differences between groups. If false: Your model finds differences between groups.

Given that you are using a model which its subyacent assumption is normallity (amongst others), the hypothesis test is to be tried comparing the probability in the context of a normal distribution.

The function stats.norm.cdf returns the probability of "lift being close to zero" if lift is supposed to be "normal". If lift is zero, then there is no difference between groups, so a p-value of <0.01 tell us that the probability that the groups are equal is almost 0, meaning that your groups are different.

The 2 is due to a concept called "two-tailed distribution": The difference between groups can be A greater than B or B greater that A, that's why you measure the difference in either two of the ways.

The addition between standard deviations obeys the concept of: $Var(X+Y) = Var(X) + Var(Y)$ if $X$ and $Y$ are independent.