# Training, cross validation and testing accuracy (RMSE and R2) differs when using different shuffles and splits

I have a very small data set of 60 observations. My training, cross-validation and testing accuracy (RMSE and R-squared) differ in a considerable amount when using different random states while performing shuffling and then splitting. The training, testing and cross validation accuracy changes each time a different random state is used. How can I solve such an issue and how to really assess the performance of the model?

• With a small dataset, "considerable" differences might still be within normal, statistically expected differences. With 60 samples and an accuracy of 70%, the 95% CI ranges from 58% to 82% accuracy, so you shouldn't be surprised if you're seeing that level of variation by resampling. – Nuclear Hoagie May 15 '20 at 16:23

In terms of evaluation, the best you can do with a very small amount of data is repeating $$k$$-fold cross-validation many times (i.e. very large $$k$$), and consider the whole distribution of scores as the performance (in particular take into account the variance across folds).