I am more familiar with classification tasks, though I have been working on a regression problem. I was given a large training dataset (>70k samples) and an independently collected test set (~2k). I consistently achieve decent validation accuracy but significantly lower accuracy on the test set.
I've been performing validation like this:
1) Standardize the training data; store the mean and variance (in an sklearn Scaler object).
2) Hold out 10% training data for validation.
3) Perform 3-fold cross validation on the remaining 90% training data. Look at the errors.
4) Train the model on the 90% training data (with 10% of this subset used as validation during training).
5) Evaluate the performance on the 10% held-out training data. This is used as my final validation measure.
Then I test my model by:
1) Standardizing the test data using the mean and variance of training data.
2) Predict and evaluate
Depending on the model I test, I typically receive 10+% drop in accuracy from validation to test. Now, to test whether this is caused by the test set coming from a different distribution, I combined the test set (~2k) with an equal-sized random portion of the training set. Then I held out 10% of this as a new test set and performed training/validation as described above using the rest. So in this case the test set is no longer an independently collected dataset.
It appears that I'm in some way overfitting to the validation set, although as I describe above, the validation portion is held out until after training.
I think it is worth noting that my target variable and many predictors are exponential-like distributed (the target variable has a lower bound at 0 and increasingly rare large values). Heteroscedasticity is apparent because plotting predicted vs. observed output shows increasing error with larger values.
I would greatly appreciate any ideas of the cause of this discrepancy. Thank you!