How to impute missing value in Test Set using a custom Imputer created on training dataset

Question

I am working on a toy project to predict claims. One of the input features has null values on which I have applied a custom imputation technique. Under this technique, I replaced missing values with the mean value of the two categories of the target feature. The code snippet is as below:

dataframe['Feature'] = dataframe['Feature'].fillna(dataframe.groupby('Target Feature')['Feature'].transform('mean'))

Using this strategy I have designed classification models based on Logistic Regression and Support Vector Classifier. Now I have to run my models on a test set but am stuck at the pre-processing stage. The test set also has missing values in the same feature (as in the training set), now how can I update these missing values with the mean values that the models learnt from the training set.

Answer 1

The way you are imputing your feature can't be replicated in the test set, because it needs knowledge of the target classes!

You need to select a different imputation strategy, that doesn't rely on your target feature.

---

Assuming that you are using another feature, the same way you were using your target, you need to store the value(s) you are imputing each column with in the training set and then impute the test set with the same values as the training set. This would look like this:

# we have two dataframes, train_df and test_df

impute_values = train_df.groupby('Another Feature')['Feature'].mean()

train_df['Feature'] = pd.Series(train_df['Feature'].values, index=train_df['Another Feature']).fillna(impute_values).reset_index(drop=True)

# train your model ...

test_df['Feature'] = pd.Series(test_df['Feature'].values, index=test_df['Another Feature']).fillna(impute_values).reset_index(drop=True)

---

Example:

train_df = pd.DataFrame({'f1': ['a'] * 5 + ['b'] * 5, 'f2': range(10)})
test_df = pd.DataFrame({'f1': ['a'] * 3 + ['b'] * 7, 'f2': range(10, 20)})
train_df.loc[[1, 6], 'f2'] = np.nan
test_df.loc[[1, 6], 'f2'] = np.nan

impute_values = train_df.groupby('f1')['f2'].mean()

train_df['f2'] = pd.Series(train_df['f2'].values, index=train_df['f1']).fillna(impute_values).reset_index(drop=True)

test_df['f2'] = pd.Series(test_df['f2'].values, index=test_df['f1']).fillna(impute_values).reset_index(drop=True)