1
$\begingroup$

I'm working in a problem in which I'm OneHot encoding a set of feaures from a dataframe, for instance:

from sklearn.preprocessing import OneHotEncoder
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y)
oh = OneHotEncoder(handle_unknown='ignore')

print(X)

      a     b     c   
1    one    m     y 
2    two    m     n  
36   three  f     n  
113  one    f     n  
31   two    m  other  
....

oh.fit(X_train)

However, it could be that not all features are present in the test set. For this example, say I only have the two first columns. The encoder, in this case will raise an error:

oh.transform(X_test.loc[:,:'b'])

The number of features in X is different to the number of features of the fitted data. The fitted data had 3 features and the X has 2 features.

Is there some way around this? Ideally I'd like to have the missing columns, either ignored (not present in the output), or set to NaN.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

It is not possible. And this has nothing to do with OHE, if you choose any other encoding methods, still you are facing the same problem! Not having similar feature sets between train and test contradicts with fundamentals of developing ML models using train/test. This is already discussed in length in a stats.stackexchange question, or here concisely.

My two cents into the matter:

  • I would first recommend figuring our the nature of missing column in the test data. Do they miss sometimes or they are never present? If they do miss sometimes, how do they look like when they are present? If they are never present in the test, you have no choice than dropping them in the train set. However, if they are present every now and then in the test, and they contain similar content to the train, maybe imputing them is kind of possible, and having a NaN in test when not present in the test is a valid approach to try. Stil I would recommend doing so, as you may add artifacts (information) that may not represent the reality or even distort the reality so to say.

  • OHE: Please note that OHE is quite sensitive to its members. As it was mentioned in one of the answers and I am sure you are aware too, OHE create new column for each of its members and fill them with 0 or 1. For example, for your column a, you will have a_one, a_two, and a_three columns after OHE. And if even one of these members (one, two, three) are not present in column a in your test set, your pipeline raises an error. OHE is good for cases that you have always a known set of members for each features, otherwise you are better good choosing other categorical encoding methods like Hashing, Binary, or even recently introduced Entity Encoding.

Good luck!

Improve this answer
$\endgroup$
9
  • 1
    $\begingroup$ Thanks for the answer, and for referencing those links. Indeed the problem I'm facing is quite unconventional. The ML model has to return a prediction in more than one step (5 or 6), and all features are not available until the end. So the question is, can a single model do?. XGB handles test sets with less features, which would allow it to return a prediction with missing features. However, the pretrained encoders are the problem. $\endgroup$
    – yatu
    Apr 9, 2020 at 8:58
  • 1
    $\begingroup$ Yes I'm aware of this last limitation you mention in the second point. Note however that by setting handle_unknown='ignore' unseen values in the test set are ignored, and no errors are raised. $\endgroup$
    – yatu
    Apr 9, 2020 at 8:58
  • 1
    $\begingroup$ One alternative could be to keep track of all columns whith which they were trained, and set those columns to dummy strings (which have not been seen in trainig) so that OneHot ignores them. However it is not very elegant, what do you think? $\endgroup$
    – yatu
    Apr 9, 2020 at 8:59
  • $\begingroup$ Interesting question about more than one step with dynamic number of features in which model needs to do inference. Here then I get to ask, does it mean that in the succeeding steps , the nature of the problem changes? I mean that if always in the succeeding steps those features are not present, and they shouldn't, then mapping between missing features to the target would be meaningless. Maybe it is best to have multiple models (like ensemble methods, though here not the model but feature space is changing). The Q you gotta ask, how hard to train separately? $\endgroup$
    – TwinPenguins
    Apr 9, 2020 at 9:26
  • 1
    $\begingroup$ So rearanging the input matrix, just as X_train is, and filling missing features with dummy values, could do the trick IMO. The only question that remains, is if the predictions will be any good $\endgroup$
    – yatu
    Apr 9, 2020 at 9:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.