Why would you split your train data to compute a value on half of the data to then fill the Nan values on the other half?

Question

I was checking a kernel written in python from the Bosch kaggle competition (kaggle link to python kernel) and I came across with a weird (at least to me) way to fill Nan values.

The train data is split into two halves and then some kind of average is computed in one half by using the non-Nan values of a field along with the target value and then fill Nan values on the other half with these computed values.

Then, when training the model after filling Nan values, the model is only trained on the half data where the Nan values have been replaced.

The question is, why would you split the data into two halves to compute on one and then fill the other? Are you introducing some kind of leakage when mean values are related to the target value and that's the reason why just the half part with filled values (the one in which you haven't computed anything, just filled Nans) is used to train? Is this procedure prone to overfitting if you perform this operation on all the train data?

Thanks in advanced.

Answer 1

You basically answered your own question:

Are you introducing some kind of leakage when mean values are related to the target value and that's the reason why just the half part with filled values (the one in which you haven't computed anything, just filled Nans) is used to train? Is this procedure prone to overfitting if you perform this operation on all the train data?

Yes. You have already used the target values and the first half of the training set. In other words, the filled-out NaN values have information about the target values in them, so if you were to perform any cross-validation for performance estimation, you would end up with an over-estimation of your performance metric. It's best practice to leave that used part out of any further training steps to avoid information leakage.

Answer 2

The reason is you (obviously) cannot feed NaNs to your model, so you absolutely have to clean them up. Looks like the idea here is – by filling them with something other than zeros may help the learning by introducing some excessive information.

It would be actually interesting first to run some sort of seq2seq RNN to try to predict NaNs from non-NaNs and then explore how it affects learning rate/accuracy...