Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I'm studying Logistic Regression for the first time.
My book explains how to fit the weights $w$ using the likelihood function $L(w)$, assuming that the individual samples in our dataset are independent of one another.
I know what independent events are. But I don't understand the meaning of the individual samples in our dataset are independent of one another in the Logistic Regression case. What does this mean? And why?
Context / big picture:
Two events are independent if they have no influence on each other's outcomes. For example, if event A is "I go get coffee" and event B is "it's raining outside", then events A and B are independent, because I'm a coffee fiend and I don't care whether it's raining - I'm getting that coffee anyway!
Logistic regression example:
Say you have blood pressure samples from 50 different individuals at a hospital. You want to classify these as high or low blood pressure. In this example, your X (input variable) is the blood pressure from the 50 individuals, and your Y (binary response) is yes/no to the question 'is this blood pressure high?'. Each person's blood pressure is independent (if we assume they are unrelated strangers).
If you were to sample the same person's blood pressure twice, like before and after physical exercise, then you would no longer have independent samples.
Why:
We need independent samples in logistic regression because otherwise the degrees of freedom of the model are not what we expect them to be, and this affects further calculations. For example, in your dataset, you have 50 independent samples. You use 2 to estimate your intercept and slope, which leaves you with 48 degrees of freedom to work with. If you didn't have independent samples, then you would not have 48 degrees of freedom.
EDIT: a bit more detail on why independent samples are important.
The assumption of independence ensures that each sample contributes the same amount of information to the experiment. With independent samples, this is the case. A blood pressure measurement from one individual is not dependent on the blood pressure of another individual - they're independent. However, if you measure the same person's blood pressure before and after exercise, a different amount of information is added. You can no longer ask the same questions as you could with independent samples.
Let's pick a simpler model to describe what's going on. We will follow standard hypothesis testing principles to illustrate the practical difference between independent and dependent samples. I will round to two decimal places for ease of reading.
Suppose we have two groups of people: those with blood type A and those with blood type O. (Aside: notice people cannot have both blood types at once). We want to investigate whether those with blood type A have a higher resting heart rate than those with blood type O. We have 12 people with blood type A and 10 people with blood type O. To answer this question, we'll conduct a two-sample hypothesis test.
Null hypothesis: The mean resting heart rate of those with blood type A is the same as the mean resting heart rate of those with blood type O
Alternate hypothesis: The mean resting heart rate of those with blood type A is different from the mean resting heart rate of those with blood type O.
We measure each person's resting heart rate, and we get the following results:
Type A: 60, 65, 70, 62, 55, 80, 70, 72, 66, 81, 77, 78
Type O: 61, 64, 70, 72, 59, 62, 66, 78, 69, 75
We then calculate the sample mean and sample standard deviation of each group:
mean(Type A) = 69.67; standard deviation(Type A) = 8.35
mean(Type B) = 67.6; standard deviation(type B) = 6.28
We can then calculate a test statistic, which will follow a t-distribution. We may decide to use either Welch's method or a pooled variance test - in this case, I'll show Welch's method, which doesn't require the population variances to be equal (unlike the pooled variance test).
The test statistic is calculated via:
standard error: sqrt((8.35^2/12) + (6.28^2/12)) = 3.02
test statistic = (69.67 - 67.6)/3.02 = 0.69
The degrees of freedom here are difficult to calculate, but a generally accepted approximation is "the lesser of the degrees of freedom in the two groups", that is min(df_1, df_2). In this case the degrees of freedom of the first group is 12 - 1 = 11 and the degrees of freedom of the second group is 10 - 1 = 9. We subtract 1 here from each sample size, because we have estimated one parameter for each group (the sample mean).
If we wanted to continue this example, we could look at a t-distribution on 9 degrees of freedom and decide based on the test statistic whether to reject or not reject the null hypothesis. In this case, we're more interested in the degrees of freedom than the result, so we'll move on.
Dependent samples
Suppose we have a that same group of people who all have blood type A. Instead of measuring their resting heart rate just once, we’ll measure each Type A person’s resting heart rate twice. This introduces dependence between the samples.
Null hypothesis: the mean resting heart rate of people with blood type A is the same as the mean resting heart rate of all blood types
Alternative hypothesis: the mean resting heart rate of people with blood type A is different from the mean resting heart rate of all blood types
We measure each person's resting heart rate before exercise, and we get the following results:
Before exercise: 60, 65, 70, 62, 55, 80, 70, 72, 66, 81, 77, 78
Then we measure their resting heart rate after exercise, and we get the following results (notice that person 1 in the previous list of heart rates is the same as person 1 in this list):
After exercise: 90, 85, 70, 72, 76, 88, 89, 92, 96, 93, 84, 85
Now we have two observations for each individual. We can’t ask the same question, unless we want to throw out half of our data! Instead, we’ll ask whether there is a difference in the resting heart rate before and after exercise. We’ll use a paired-difference t-test.
To conduct this test, we have to take both observations from each individual into account. We calculate the differences between each person’s before and after measurement, and treat this set of differences as our actual data set:
60-90, 65-85,70-70, 62-72, …
Then we can calculate a sample mean, a sample standard deviation, etc. Our degrees of freedom would be 12 -1 = 11.
What would have happened if we had assumed that those dependent samples were independent?
Notice that when we had dependent samples, our degrees of freedom was 11; when we had independent samples, our degrees of freedom was 9. If we had assumed that the dependent samples were in fact two independent samples, then we could have used the wrong degrees of freedom.
** What if we had had only 10 measurements instead of 12 in the ‘dependent samples’ section? Wouldn’t we have had the same degrees of freedom? **
Yes, but it still wouldn’t have meant the same thing. We would have had:
Before exercise: 60, 65, 70, 62, 55, 80, 70, 72, 66, 81, 77, 78
After exercise: 90, 85, 70, 72, 76, 88, 89, 92, 96, 93
And we would have had to ‘throw away’ the last two ‘Before exercises’ measurements and use only the first ten, as the last two had no partner measurement. The meaning has still changed, though. With dependent samples, you cannot ask the same kinds of questions as with independent samples.
Each sample (read: each row in your table/matrix) can be seen as the realization of an $m$-dimensional random variable $X_{i}$, where $m$ is the number of features (read: number of columns in your table/matrix).
Independence means that the random variables $X_i, i = 1, \dots, n,$ that generate your samples are independent. StatsSorceress explains the intuition behind independence wonderfully.
A different way of looking at it is: 'Does knowing sample $X_i$ give me additional information about another sample $X_j$?' One case where this is true (and the samples are not independent) is for time series. If your samples are stock prices, and you know that the stock price one second ago was USD 100, then you can be fairly sure that the stock price right now is also going to be close to USD 100.
