How to make predictions on unseen data with different cardinality using xgboost

Question

I am training an XGBoost regression model on a feature set $X$ that includes a feature $x_k$ with high cardinality (~100). First, I am using one-hot-encoding to convert $x_k$ and then split the set into training and testing sets. The model seems to work well. So far everything's pretty standard.

The problem arises when I make predictions on unseen data. In the unseen data, the cardinality of $x_k$ is slightly different. To put into perspective, say the unique values of $x_k$ in training & testing set was $\lbrace aa,ab,ac,...,ay,az \rbrace$. In the unseen data, the cardinality set is $\lbrace aa, ab, ...., ay \rbrace $. So $az$ does not appear in unseen data. To be able to make predictions with the model I had, I have to have the same columns in the unseen data as in training. To remedy this, I tried two ways:

1. Create a new column in unseen data that will correspond to the dummy variable $az$, and assign 0 to all rows. Here, my logic was: "OK, apparently $az$ is observed rarely so creating a column with all zero values would make the sets consistent and should not hurt accuracy so bad". It did.
2. Train the XGBoost model by deleting the column corresponding to the dummy variable $az$. You can think of it as applying one-hot-encoding with a cardinality set $\lbrace aa, ab, ...., ay \rbrace $. In this case, the accuracy in testing set and on unseen data were good. However, the problem is I needed to train the model again, and I do not want to do so each time I want to make predictions.

I know there are methods other than one-hot-encoding for categorical features, but I am curious on how to approach to this specific issue.

Answer 1

There isn't much to philosophize here, in order for your model to predict the value for some class, it has to see that class in training. If the model never saw that class it cannot predict it. So you should make sure that all classes are present in your training set. If a new class appears at a later time you have to retrain your model with the new class.

Having said that, a workaround could be to have a reference class, such as "other", where you would put all classes that are not worthy of having their own class, or new classes, and used that class for any new classes which were not present in training. However, you can expect the results to be not great.