# Training, cross validation and testing accuracy (RMSE and R2) differs when using different shuffles and splits

I have a very small data set of 60 observations. My training, cross-validation and testing accuracy (RMSE and R-squared) differ in a considerable amount when using different random states while performing shuffling and then splitting. The training, testing and cross validation accuracy changes each time a different random state is used. How can I solve such an issue and how to really assess the performance of the model?

• With a small dataset, "considerable" differences might still be within normal, statistically expected differences. With 60 samples and an accuracy of 70%, the 95% CI ranges from 58% to 82% accuracy, so you shouldn't be surprised if you're seeing that level of variation by resampling. May 15, 2020 at 16:23

## 2 Answers

In terms of evaluation, the best you can do with a very small amount of data is repeating $$k$$-fold cross-validation many times (i.e. very large $$k$$), and consider the whole distribution of scores as the performance (in particular take into account the variance across folds).

It's going to be difficult anyway to obtain a reliable measure of performance with such a small dataset. Two options come to mind:

• obtain more instances, possibly by using some interpolation method to generate artificial data (but it's not as good as real data).
• make the model less complex by reducing the number of features, as this is likely to reduce the variance in the performance.

Take a look at this: https://stats.stackexchange.com/questions/335936/choosing-the-correct-seed-for-reproducible-research-results?fbclid=IwAR1i1-WjSYxCQrV5GU5-LHD6rU7VYfoE_X-xg3J7zmQa2o2Obnf27CDfwuY there is a very thorough answer that might be of use.