Suppose these two cases:
- a. I have access to 100 examples. I then have access to another batch of 100 examples.
- b. I have access to 200 examples, in once.
I am reading that in the first, I could do a T-test to check type-error I, by comparing stats analysis against the two independent datasets. But I could not in the second case, because if I was to split those 200 examples in two datasets, they would not be independent.
Ref: S.B. Kotsiantis, "Machine Learning: A Review of Classification Techniques", 2007
In practice, however, we often have only one dataset of size N and all estimates must be obtained from this sole dataset. Different training sets are obtained by subsampling, and the instances not sampled for training are used for testing. Unfortunately this violates the independence assumption necessary for proper significance testing. The consequence of this is that Type I errors exceed the significance level.
- Question 1 Imagine I shuffle the data in .b and split them in half, what woudl be the difference I did that instead of another person I don't know about, that is the one that provided me with that data ?
In other words, why in the second case the two partitions are not independent, even if both cases I have same assumption that both datasets come from the same "distribution of chances" for having been picked up in observing a phenomena ?
- Question 2 The second question, is if you can help elaborating on type I error and its relevance here (I am not familiar with it) : can you help understand with practical example, referring to "Type-I error exceeds the significance level" (see quoting in bold)?
