My training data has no missing values. I'm unsure whether or not I should fit say, imputation, on the training set so that I can accommodate possible missing values on the test set, because the test set is 'locked away' during training time. Should I peek at the test data to see if there are missing values, or should I fit imputation for all features on a complete training set?
First, we must understand about a common statistical term called
population. Given a population say
random sample is drawn (in the ideal conditions). Now suppose you are asked to build a predictive model based on this random sample. So, you split the sample into train, test and validation sets. And you start to build the model on the train set. You begin with the initial data cleaning activities, and you find there is no missing data in it.
With this brief background, now to answer your Q literally forces me to rather ask you a question;
Q. Did you collect the data or was it given to you?
If your response to the former part of my Q is
yes then by all means do whatever you want with the data. And if your response to the latter part of my Q is
yes then you should not make any attempt to pollute it. Nor should you make any attempt to peek into the holdout dataset, because then subconsciously the predictive model you have built will be sabotaged.
Finally, remember the concept of population and sample. It holds the key. Build your model such that it accounts for all initial data preprocessing activities. If a particular step does not fulfill the dataset, then code can be written to avoid it
Basis of the OP suggesting the dataset was given to them, I'm revising the answer further;
i think your getting confused between the population data and new data. As per my answer, models are built basis of the population data. The new data should have the same attributes and properties as the population data. However, if for some reason unknown to you, the new data has missing values but the original data does not, then there are a couple of options, namely;
a. Determine the missing data pattern,
i. if its missing completely at random (MCAR), means there is no relationship between the missingness of the data and any values, observed or missing data. In other words, no systematic differences exist between participants with missing data and those with complete data. In these instances, the missing data reduce the analyzable population of the study and consequently, the statistical power, but do not introduce bias: when data are MCAR, the data which remain can be considered a simple random sample of the full data set of interest.
ii. Missing at random (MAR). When data are MAR, the fact that the data are missing is systematically related to the observed but not the unobserved data.
iii. Missing not at random (MNAR). When data are MNAR, the fact that the data are missing is systematically related to the unobserved data, that is, the missingness is related to events or factors which are not measured by the researcher.
Once you have determined the nature of missingness, then you should determine if the missing data can be imputed or not.
In conclusion, if the new data has missing values then you can treat them or else remove them. The choice is yours. My suggestion will be to follow the middle path, in which you can build 2 models; first model without the missing data and second model with the imputed missing data. Evaluate and validate both the model performance and choose the one which closely matches the desired business/research outcome.
If the Train data(~80%) doesn't have any missing records and you are expecting missing records in test data(~20%).
This can happen in these circumstances(can be other too) -
Only few missing records in the count -
Then these are most probably completely at random, then you can either remove the records or fill with the mean/median of training data
A Good number of missing records in the count -
It means the Training set is not representing the dataset properly. Then anyway your model will struggle with this Test data and you might have to create a better Train/Test set.
Also, chances are high that missingness will fall under the other two Categories. So you may impute accordingly.