I have a very small data set of 60 observations. My training, cross-validation and testing accuracy (RMSE and R-squared) differ in a considerable amount when using different random states while performing shuffling and then splitting. The training, testing and cross validation accuracy changes each time a different random state is used. How can I solve such an issue and how to really assess the performance of the model?
1
-
1$\begingroup$ With a small dataset, "considerable" differences might still be within normal, statistically expected differences. With 60 samples and an accuracy of 70%, the 95% CI ranges from 58% to 82% accuracy, so you shouldn't be surprised if you're seeing that level of variation by resampling. $\endgroup$– Nuclear HoagieCommented May 15, 2020 at 16:23
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
In terms of evaluation, the best you can do with a very small amount of data is repeating $k$-fold cross-validation many times (i.e. very large $k$), and consider the whole distribution of scores as the performance (in particular take into account the variance across folds).
It's going to be difficult anyway to obtain a reliable measure of performance with such a small dataset. Two options come to mind:
- obtain more instances, possibly by using some interpolation method to generate artificial data (but it's not as good as real data).
- make the model less complex by reducing the number of features, as this is likely to reduce the variance in the performance.
$\begingroup$
$\endgroup$
Take a look at this: https://stats.stackexchange.com/questions/335936/choosing-the-correct-seed-for-reproducible-research-results?fbclid=IwAR1i1-WjSYxCQrV5GU5-LHD6rU7VYfoE_X-xg3J7zmQa2o2Obnf27CDfwuY there is a very thorough answer that might be of use.
