I have a model that is based on an experiment collected on 100 subjects. We are testing the model as follows:

1. Record raw data from the subjects
2. For each subject, compute the feature from the raw data (thus, we obtain 100 datasets with 25 features)
3. Reserve the data of one subject for model validation
4. Combine the remaining 99 datasets into one large dataset (thus, we obtain a dataset with up to 1 million rows and 25 columns)
5. The obtained large dataset is used to train a model and evaluate it using a 10-folds cross-validation.
6. Use the reserved dataset (see step 3) to validate the model obtained in step 5 above

Unfortunately, the model's performance is very confusing to me (is this an overfitting?) :

1. The cross-validated model achieves 99.9% accuracy and 99.7% recall)
2. However, when the same model is tested using the validate test set (see step 3), I get a very low accuracy (40.2% recall and 39.8% precision)

What could be the reason for this discrepancy? Any suggestions on how this could be improved?

I have a model that is based on an experiment collected on 100 subjects. We are testing the model as follows:

1. Record raw data from the subjects
2. For each subject, compute the feature from the raw data (thus, we obtain 100 datasets with 25 features)
3. Reserve the data of one subject for model validation
4. Combine the remaining 99 datasets into one large dataset (thus, we obtain a dataset with up to 1 million rows and 25 columns)
5. The obtained large dataset is used to train a model and evaluate it using a 10-folds cross-validation.
6. Use the reserved dataset (see step 3) to validate the model obtained in step 5 above

Unfortunately, the model's performance is very confusing to me (is this an overfitting? a data leakage?) :

1. The cross-validated model achieves 99.9% accuracy and 99.7% recall)
2. However, when the same model is tested using the validate test set (see step 3), I get a very low accuracy (40.2% recall and 39.8% precision)

What could be the reason for this discrepancy? Any suggestions on how this could be improved?