I have a very small data set of 60 observations. My training, cross-validation and testing accuracy (RMSE and R-squared) differ in a considerable amount when using different random states while performing shuffling and then splitting. The training, testing and cross validation accuracy changes each time a different random state is used. How can I solve such an issue and how to really assess the performance of the model?
In terms of evaluation, the best you can do with a very small amount of data is repeating $k$-fold cross-validation many times (i.e. very large $k$), and consider the whole distribution of scores as the performance (in particular take into account the variance across folds).
It's going to be difficult anyway to obtain a reliable measure of performance with such a small dataset. Two options come to mind:
- obtain more instances, possibly by using some interpolation method to generate artificial data (but it's not as good as real data).
- make the model less complex by reducing the number of features, as this is likely to reduce the variance in the performance.
Take a look at this: https://stats.stackexchange.com/questions/335936/choosing-the-correct-seed-for-reproducible-research-results?fbclid=IwAR1i1-WjSYxCQrV5GU5-LHD6rU7VYfoE_X-xg3J7zmQa2o2Obnf27CDfwuY there is a very thorough answer that might be of use.