I am working on a toy project to predict claims. One of the input features has null values on which I have applied a custom imputation technique. Under this technique, I replaced missing values with the mean value of the two categories of the target feature. The code snippet is as below:

dataframe['Feature'] = dataframe['Feature'].fillna(dataframe.groupby('Target Feature')['Feature'].transform('mean'))

Using this strategy I have designed classification models based on Logistic Regression and Support Vector Classifier. Now I have to run my models on a test set but am stuck at the pre-processing stage. The test set also has missing values in the same feature (as in the training set), now how can I update these missing values with the mean values that the models learnt from the training set.


1 Answer 1


The way you are imputing your feature can't be replicated in the test set, because it needs knowledge of the target classes!

You need to select a different imputation strategy, that doesn't rely on your target feature.

Assuming that you are using another feature, the same way you were using your target, you need to store the value(s) you are imputing each column with in the training set and then impute the test set with the same values as the training set. This would look like this:

# we have two dataframes, train_df and test_df

impute_values = train_df.groupby('Another Feature')['Feature'].mean()

train_df['Feature'] = pd.Series(train_df['Feature'].values, index=train_df['Another Feature']).fillna(impute_values).reset_index(drop=True)

# train your model ...

test_df['Feature'] = pd.Series(test_df['Feature'].values, index=test_df['Another Feature']).fillna(impute_values).reset_index(drop=True)


train_df = pd.DataFrame({'f1': ['a'] * 5 + ['b'] * 5, 'f2': range(10)})
test_df = pd.DataFrame({'f1': ['a'] * 3 + ['b'] * 7, 'f2': range(10, 20)})
train_df.loc[[1, 6], 'f2'] = np.nan
test_df.loc[[1, 6], 'f2'] = np.nan

impute_values = train_df.groupby('f1')['f2'].mean()

train_df['f2'] = pd.Series(train_df['f2'].values, index=train_df['f1']).fillna(impute_values).reset_index(drop=True)

test_df['f2'] = pd.Series(test_df['f2'].values, index=test_df['f1']).fillna(impute_values).reset_index(drop=True)

  • $\begingroup$ impute_values is a Series now. It will work for the train because the size is same but it will not work for the test as it simply fills based on the Index. If train size is greater than the test, it will fill incorrect values $\endgroup$
    – 10xAI
    Commented Jan 18, 2021 at 14:16
  • 1
    $\begingroup$ @10xAI you are right, I fixed it now $\endgroup$
    – Djib2011
    Commented Jan 21, 2021 at 9:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.