Overfitting with sklearn pipeline - reasons why?

Question

So....

I've been playing around with this for FAR TOOO LONG now and I really need some advice.

Most people on kaggle concat training and testing set TOGETHER and then pre scale the data, this seems to provide good results when scoring but i believe this to be data leakage and an incorrect process to perform in real world solutions.

I've kept apart 15% of data as a holdoutset and the rest for CV.

The issue I have is that if I put the scaler into a pipeline, the scale changes on each fold of the CV and gives me wildly different results to if I prescaled the whole dataset prior to hyper parameter tuning.

I'm even getting worse results once I fit to the whole training data (including the holdout set) which suggests its overfitting to the smaller data set.

Is there anything I can do to combat this?

Am I doing something wrong? (should I be putting a scaler into a pipeline?)

Or is this just what it is in the real world?

Any other help regarding overfitting etc would be amazing.

Answer 1

If your dataset is giving really different results for different folds and you just have a scaler and a model in your pipeline could be for different reasons:

• The splits of your validation have different difficulties for some reason. Some are just easy to predicto of you haven't choosen the proper validation and crossfold. An example of this would be that there is a temporal dependency and you are not getting it. If this was the case my reccomendation will be to choose a proper validation and crossfold.

• Other possible problem would be that the features are not equally distributed, meaning that there are some outliers. If there is outliers when you scale you will have different intervals. If this was the case I would say that before doing anything more complicated doing some pre processing for the whole dataset.

Note: Doing what Kagglers do of concateneting train and test can be dangerous when you put your model into production.

Answer 2

Ok first things first, I do not think it's a good idea to concatenate train and test set for anything and you are right in stating the problem of data leakage.

Now as to getting different results when using scaler in cv is expected. This is because, for every iteration of cv, the data set changes. For example if cv = 3, then for the first iteration, it will consider a different train and valid set, for second iteration, the train and valid set will be different and so on. I used to get different values of MAE for each iteration in my RandomForest model if I performed CV.

As for the solution I can suggest doing the following:

Scale the train set and the test set before fitting or doing anything. Also whatever preprocessing you do to your train set, do the exactly same using the same parameters for your test set to avoid leakage!

Answer 3

As stated in the earlier comments, preprocessing both train and test set at the same time causes serious generalization error in the real life applications. So, I totally agree with you at this point.

When it comes to scaler issue, the first thing that I came up with is that you can evaluate the importances of scaled features over the target value. Then, you can discretize these features based on their distributions and histogram plot (such as if age > 18 then age_cat 1 etc.) by using all training data, and stratified sampling for cross validation folds based on these features (age_cat). It may provide better value distributions among each fold. The only thing that you should consider is the outliers, they may distort scaler as you know.

Good luck!