1
$\begingroup$

I have a very small data set of 60 observations. My training, cross-validation and testing accuracy (RMSE and R-squared) differ in a considerable amount when using different random states while performing shuffling and then splitting. The training, testing and cross validation accuracy changes each time a different random state is used. How can I solve such an issue and how to really assess the performance of the model?

$\endgroup$
1
  • 1
    $\begingroup$ With a small dataset, "considerable" differences might still be within normal, statistically expected differences. With 60 samples and an accuracy of 70%, the 95% CI ranges from 58% to 82% accuracy, so you shouldn't be surprised if you're seeing that level of variation by resampling. $\endgroup$ May 15 '20 at 16:23
1
$\begingroup$

In terms of evaluation, the best you can do with a very small amount of data is repeating $k$-fold cross-validation many times (i.e. very large $k$), and consider the whole distribution of scores as the performance (in particular take into account the variance across folds).

It's going to be difficult anyway to obtain a reliable measure of performance with such a small dataset. Two options come to mind:

  • obtain more instances, possibly by using some interpolation method to generate artificial data (but it's not as good as real data).
  • make the model less complex by reducing the number of features, as this is likely to reduce the variance in the performance.
$\endgroup$
0
$\begingroup$

Take a look at this: https://stats.stackexchange.com/questions/335936/choosing-the-correct-seed-for-reproducible-research-results?fbclid=IwAR1i1-WjSYxCQrV5GU5-LHD6rU7VYfoE_X-xg3J7zmQa2o2Obnf27CDfwuY there is a very thorough answer that might be of use.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.