7
$\begingroup$

The SimpleImputer class takes pandas dataframes and returns unlabeled numpy arrays. Which means that the SimpleImputer drops some features at will, but has no way to communicate which features have been dropped to the caller

I've been trying to come up with a workaround, but they all are extremely hackish and unreliable. Is there something I'm missing?

$\endgroup$
5
$\begingroup$

I've got the same issue today, and it's a shame your post got no answers. I think this question is not well addressed in the sklearn documentation. I can show you my workaround to this issue:

headers = X.columns.values
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

empty_train_columns =  []
for col in X_train.columns.values:
    # all the values for this feature are null
    if sum(X_train[col].isnull()) == X_train.shape[0]:
        empty_train_columns.append(col)
print(empty_train_columns)

The idea is to keep all your column names, and after you split your data check which of them completely empty in your training set. If I'm not wrong the Imputer respects the column order so, for example, you can correlate every feature with its importance if you are using Decision-Tree-based models.

I'm not satisfied with this ugly piece of code but I couldn't find a more elegant (and simple) solution.

$\endgroup$
0
$\begingroup$

SimpleImputer drops columns consisting entirely of missing values. It is indeed unpleasant when trying to associate original columns; the sklearn devs have been discussing this:

https://github.com/scikit-learn/scikit-learn/issues/16426

Vincent's answer is good, if you are working directly: just detect and remove the offending all-missing columns, since they don't contribute anything to your model. If you need something more automatic (e.g., you have a highly-missing column that in cross-validation leads to some training folds being all-missing), then perhaps use a ColumnTransformer, where the columns argument is a callable that checks for all-missing? Then you can use the ColumnTransformer's get_feature_names method to find out when/if a column was removed.

$\endgroup$
0
$\begingroup$

It's a really easy problem. Suppose you want to fit an imputer to your X_train and test data and keep the column names of X in the imputed results:

imp = SimpleImputer()
X_train = pd.DataFrame(imp.fit_transform(X_train), columns = X.columns)
X_test = pd.DataFrame(imp.transform(X_test), columns = X.columns)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.