I've been playing around with this for FAR TOOO LONG now and I really need some advice.
Most people on kaggle concat training and testing set TOGETHER and then pre scale the data, this seems to provide good results when scoring but i believe this to be data leakage and an incorrect process to perform in real world solutions.
I've kept apart 15% of data as a holdoutset and the rest for CV.
The issue I have is that if I put the scaler into a pipeline, the scale changes on each fold of the CV and gives me wildly different results to if I prescaled the whole dataset prior to hyper parameter tuning.
I'm even getting worse results once I fit to the whole training data (including the holdout set) which suggests its overfitting to the smaller data set.
Is there anything I can do to combat this?
Am I doing something wrong? (should I be putting a scaler into a pipeline?)
Or is this just what it is in the real world?
Any other help regarding overfitting etc would be amazing.