2
$\begingroup$

The usual workflow when building a machine learning model starts with random splitting the data set into training and test set.

What I can't understand is why we do this. For example lets say we have a labeled data set of 100 data points: $$\{x_i, y_i\}_{i=1}^{100}$$

and the $x$ values are distributed in the following way:

$$x \in[0, 5) \rightarrow 80 \, \, \text{samples}$$

and

$$x \in[5, 10) \rightarrow 20 \, \, \text{samples}$$

Then if we random sample 80 data points to generate our training how do we know that we won't pick the 80 samples from the first interval?

I mean it is completely possible as all data points are equiprobable during sampling. And if that happens then our model wouldn't be able to see "trends" in regions outside of $[0, 5)$. Why then random sampling is the way to go when we do train-test split?

$\endgroup$
1
  • $\begingroup$ Most of my models do not do a random split. My models tend to deal with time. My training might be from one time frame, validation from another. Whenever I have a model or other statistic that needs a random split, I repeat the analysis multiple times with different splits to ensure I have stability. $\endgroup$
    – Craig
    Jun 27 at 10:10

1 Answer 1

2
$\begingroup$

Then if we random sample 80 data points to generate our training how do we know that we won't pick the 80 samples from the first interval?

Because this is extremely unlikely: one can calculate the probability to include $N$ instances with $x>5$ when every instance has an 80% chance to be selected for the training set, for any $N$. The most likely $N$ would be $N=20\times 0.8 = 16$, with probability decreasing far from this value: e.g. $N=15$ or $N=17$ is still quite likely, but for instance $N=8$ is very unlikely and $N=0$ extremely unlikely.

Btw this is an interesting experiment to run in order to understand statistics: loop over this random sampling, and see how many times you get every value of $N$ as a result.

Of course the main reason why it's recommended to randomly split the training/test set is that the instances might not have been collected randomly.

$\endgroup$
2
  • $\begingroup$ To get the probability of all samples being in a specific interval I must take the desired combinations and divide by the total combinations right? $\endgroup$
    – ado sar
    Jun 27 at 19:39
  • $\begingroup$ @adosar yes, this is correct. In the special case of picking only the 80 instances with x<5, the probability should be equal to 0.8^80 if I'm not mistaken. $\endgroup$
    – Erwan
    Jun 27 at 21:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.