Cross-fold validation done on whole dataset or training set?

Question

I have a dataset of 77 samples with 302 features with two labels (0,1). I trained an SVM with gridsearch (cv=5) to perform binary classification.

In one run of my script, I do a test-train split, and then pass the training data into gridsearch. I am getting wide range of training accuracy (>70-90%) and test accuracy (40-75%). My question, is how do I evaluate my model if it appears that based on the train/since it seems like my data is getting different results based on the splitting.

What I've done is: run the entire scrip in a loop and iterate over 10 different splits and then plot the error bars on the ROC curve. Is this reasonable?

Answer 1

Rather than a train-test split, I would recommend n-fold cross-validation or, better yet, leave-one-out cross-validation.

My question, is how do I evaluate my model if it appears that based on the train/since it seems like my data is getting different results based on the splitting.

The high variance in your error estimates is a consequence of a very small dataset. When sampling from such a small dataset, it's super easy to wind up with train and test splits that aren't representative. Cross-validation will give you a more stable error estimate.

What I've done is: run the entire scrip in a loop and iterate over 10 different splits and then plot the error bars on the ROC curve. Is this reasonable?

It's a very good idea to run your model evaluation routine several times and average the results. It sounds to me like you're manually running a less-controlled version of cross-validation.

I'm not sure what you mean when you say "plot the error bars on the ROC curve". Is area-under-ROC your accuracy metric? Or are you computing the accuracy then somehow plotting that on the same axes as an ROC curve?

---

This wasn't part of your question, but it may be helpful nonetheless. When you have a dataset with more features than examples, then you are incredibly likely to run into problems related to the curse of dimensionality. In the 302-dimensional space where your features reside, your dataset of just 77 examples is incredibly sparse.

You may want to consider simpler models (logistic regression, shallow decision trees, etc) and employ some type of regularization to discourage your model from learning spurious relationships. You could also try to use a dimensionality reduction technique to eliminate some of your 302 features.

Answer 2

do bagging and then evaluate model,

also you should split your data into train and test 60/40 and on train do folds or split it into train and validation

https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets

problem with your dataset is that it's small so maybe just use logistic regression with bagging

Answer 3

Data size is too small compared to features. Ideal case of underfitting. Please do the following...

1. Try to get more data if fesible

2. Do feature selection to narrow down features

3. Use decision tree as data size is small