3
$\begingroup$

I am more familiar with classification tasks, though I have been working on a regression problem. I was given a large training dataset (>70k samples) and an independently collected test set (~2k). I consistently achieve decent validation accuracy but significantly lower accuracy on the test set.

I've been performing validation like this:
1) Standardize the training data; store the mean and variance (in an sklearn Scaler object).
2) Hold out 10% training data for validation.
3) Perform 3-fold cross validation on the remaining 90% training data. Look at the errors.
4) Train the model on the 90% training data (with 10% of this subset used as validation during training).
5) Evaluate the performance on the 10% held-out training data. This is used as my final validation measure.

Then I test my model by:
1) Standardizing the test data using the mean and variance of training data.
2) Predict and evaluate

Depending on the model I test, I typically receive 10+% drop in accuracy from validation to test. Now, to test whether this is caused by the test set coming from a different distribution, I combined the test set (~2k) with an equal-sized random portion of the training set. Then I held out 10% of this as a new test set and performed training/validation as described above using the rest. So in this case the test set is no longer an independently collected dataset.

It appears that I'm in some way overfitting to the validation set, although as I describe above, the validation portion is held out until after training.

I think it is worth noting that my target variable and many predictors are exponential-like distributed (the target variable has a lower bound at 0 and increasingly rare large values). Heteroscedasticity is apparent because plotting predicted vs. observed output shows increasing error with larger values.

Here are results using an MLP regressor with the original training data and test data (Left: Validation, Right: Test): enter image description here

Here are results using an MLP with equally combined train/test (~4k) as training, with 10% held out for testing (as described) (Left: Validation, Right: Test): enter image description here

Here are results using a Random Forest with same train/test as for the figure directly above (Left: Validation, Right: Test): enter image description here

I would greatly appreciate any ideas of the cause of this discrepancy. Thank you!

Improve this question
$\endgroup$
4
  • $\begingroup$ You didn't mention the result of your experiment testing whether the test set comes from a different distribution. $\endgroup$
    – Ben Reiniger ♦
    Commented Aug 11, 2019 at 21:54
  • $\begingroup$ @Ben Reiniger I didn't perform a statistical test to see if the test set came from a different distribution, instead, as described, I combined the test set with an equal portion of training data as a new training set, holding out 10% of this mixed set as the new test set. The lower two figures show results using this mixed data. I did this to see if I would still notice a drop from validation to test after using some of the test data in training. $\endgroup$
    – jack
    Commented Aug 11, 2019 at 22:01
  • $\begingroup$ Ah, sorry, hadn't noticed that in the figure descriptions. At first glance, nothing stands out... $\endgroup$
    – Ben Reiniger ♦
    Commented Aug 11, 2019 at 22:14
  • $\begingroup$ @Ben Reiniger The last figure is a particularly confusing example to me, because though the validation and test were both sampled and held out from the same mixed training dataset, the performance is very different. See my comment to David Waterworth's response. $\endgroup$
    – jack
    Commented Aug 11, 2019 at 23:00

1 Answer 1

Reset to default
1
$\begingroup$

I don't think this is ususual, I experiance it quite frequently with regression problems. Generally I think it means the model is underspecified so instead of learning the actual relationship between X and y it overfits. Generally I find that I have to do a bit of feature engineering, and ensure I'm only using features that are really necesary - too many features or features with spurious relationships to the target can really hurt a RF model in particular in my experiance. Also make sure as part of your cross validation that you're cross validating parameters which directly influence model complexity i.e. tree depth, number of trees etc.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Thank you, I agree further feature engineering could be useful. Though, I'm still confused as to how, for example, in the last figure, the random forest fit quite well to a validation set (held out from training) though poorly to a test set. In that figure, I had created a 50/50 mix of the original training and test data, and used that for training, validation and testing. Thus validation and testing data were sampled from the same distribution. The only difference being that validation was separated from training data after standardization, but test was standardized using training mean & var $\endgroup$
    – jack
    Commented Aug 11, 2019 at 22:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.