NOTE: This question was first posted on a different SO forum but I received suggestions to move it here
This is a follow-up question to a question I had previously posted on this forum We conducted an experiment on 100 subjects and obtained a dataset that was used to train a machine learning model that works almost perfectly (we get a 99.99% accuracy) using cross-validation but fails miserably when test to new unseen data (we get only 39.8% accuracy). In detail, here is how we are testing our model:
- Conduct an experiment (100 subjects) and recording physiological data (skin conductance and heart rate)
- For each subject, compute the feature from the raw data (thus, we obtain 100 datasets with 25 features)
- Reserve the whole data of one subject for model validation
- Combine the remaining 99 datasets into one large dataset (thus, we obtain a dataset with up to 1 million rows and 25 columns)
- The obtained large dataset is used to train a model and evaluate it using a 10-folds cross-validation.
- Use the reserved dataset (see step 3) to validate the model obtained in step 5 above
The model performs very well (99.9% accuracy) when we used cross-validation (step 5) but fails to generalize (only 40% accuracy) when tested on new data from the unseen subject (step 3).
After reading the comments I received in my previous question, I decided to revise how I validate the model. Thus, instead of removing all the sample of one subject (as I did in step 3), I only remove all but a few samples (just 0.1% of the sample, i.e around 100 samples in my case). Surprising the model performs significantly better and achieves a 90% accuracy. This is puzzling and I would like to ask a few questions:
(1)How can one explain that a few samples (just 100 samples) are responsible for such an increase in performance (from 40% to 90% accuracy)(I will need to publish my result in a reputable conference)?
(2) Is there any published work that discusses this type of phenomena?