I have a very small dataset with only 70 samples an 18 features. The issue is that in this type of data set, splitting the data set into two parts, training and testing, causes a series of important information to be deleted. In your opinion, how should I evaluate this dataset because common methods such as dividing the dataset into test and training cannot help me?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
3
This is a common problem:
- Evaluation on test data (i.e. data not seen during the training) is crucial, especially with few samples, the rist of overfitting is significant and unseen data helps to detect overfitting.
- For a statistically meaningful (i.e significant) evaluation, you quite some test samples.
- There are so few samples available that putting some for testing aside will cause serious problems for the training. Especially when you need nearly all for significant test results.
The solution
There is a computational expensive solution that I would suggest, here: leave-one-out-testing.
It is done by training 70 models, each on 69 samples. Then use one sample, that was left out, to get a prediction for the one unseen sample. By doing it 70 times, you will get a prediction each of your 70 samples; always from a model, that was not trained in said sample. You can then compute your evaluation metrics (accuracy, $r^2$, roc-auc,...) from these 70 predictions and the corresponding lables of the 70 samples.
The idea behind it, that a model trained on 69 samples will not differ much from a model trained on 70 samples - except that it does not know on the one sample and, hence, cannot overfit on it. This allows to get a good approximation how 70 new, unseen samples would perform on a model, trained on all 70 samples.
There is even support for this in some frameworks like scikit-learn
The downside: you need to train 70 models, which can be computationally expensive.
-
$\begingroup$ Leave one out is an extreme example of k-fold cross validation. en.wikipedia.org/wiki/…. Although "you need to train 70 models, which can be computational[ly] expensiv[e]" is obviously not true today. $\endgroup$– ValentasCommented Aug 24 at 4:55
-
$\begingroup$ While leave-one-out can be seen as an extreme k-fold, it requires a slightly different way of evaluation (see also this answer). This is not part of the wiki article that you linked, because they just consider the mean squared error as evaluation metric. For metrics, like $r^2$ or roc-auc, one one must compute the metric once over all predictions. $\endgroup$– BroeleCommented Aug 24 at 13:09
-
1$\begingroup$ While I would agree with you on the computational time in most cases (especially with tabular data of 18 columns), but with the aim the write answers that might be helpful to others as well, a "can be computationally expensive" (btw: thanks for finding the typos) seems to be a reasonable warning. $\endgroup$– BroeleCommented Aug 24 at 13:15