2
$\begingroup$

I have a model that is based on an experiment collected on 100 subjects. We are testing the model as follows:

  1. Record raw data from the subjects
  2. For each subject, compute the feature from the raw data (thus, we obtain 100 datasets with 25 features)
  3. Reserve the data of one subject for model validation
  4. Combine the remaining 99 datasets into one large dataset (thus, we obtain a dataset with up to 1 million rows and 25 columns)
  5. The obtained large dataset is used to train a model and evaluate it using a 10-folds cross-validation.
  6. Use the reserved dataset (see step 3) to validate the model obtained in step 5 above

Unfortunately, the model's performance is very confusing to me (is this an overfitting? a data leakage?) :

  1. The cross-validated model achieves 99.9% accuracy and 99.7% recall)
  2. However, when the same model is tested using the validate test set (see step 3), I get a very low accuracy (40.2% recall and 39.8% precision)

What could be the reason for this discrepancy? Any suggestions on how this could be improved?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ You could try to analyze the error of predicting the unseen data using visualizations. $\endgroup$
    – Franziska W.
    Commented Jan 3, 2019 at 19:44
  • $\begingroup$ How do you visualize a dataset with 25 features? $\endgroup$
    – Lapatrie
    Commented Jan 4, 2019 at 0:16
  • 1
    $\begingroup$ Plot each feature of the unseen data vs. the prediction error. This way you may be able to see how and which features are involved in the overfitting. $\endgroup$
    – Franziska W.
    Commented Jan 4, 2019 at 6:03

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.