1
$\begingroup$

I found this very informative figure, on how to split the dataset depending on how much data (or how many observations to be more precise) you have.

My question is, since "less data" is very subjective, is there a statistical test you can perform or even a rule of thumb on which split to follow?

My current problem is a classification problem with 145 observations, 22 features, 2 labels (18-True, 127-False), but I'm interested on the general approach.

Thank you.


enter image description here

$\endgroup$
0
$\begingroup$

As usual in these cases, there is no magic wand to determine which splitting method to use. It all depends on your specific data. Is your collected data redundant enough so that k-fold cross validation is not necessary? Does your data capture most of the input space?

Now, taking a look at your numbers, I'd say that the number of observations you have (145) is not likely to be large enough to capture all the potential variability in the input space, giving the fact that you have a high number of features (22). This conclusion, of course, depends on the type of the features (are they binary? categorical? numerical?), and whether all these features are actually necessary to make a prediction (are there redundant/correlated features? features that do not give any information about the output variables?).

In your case, and not knowing more about the data, I'd go for a cross-validation splitting scheme.

$\endgroup$
1
  • $\begingroup$ Thank you @pablo for your reply. I am aware that there are no magic wand but I would expect a statistical test to validate if your "partitioning" is good. For example, my intuition says that the variance between the scores of the runs of k-fold might be a good indication. Something like this. $\endgroup$ – Yannis Mar 10 '17 at 12:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.