I have structured data with lots (tens of thousads) of categories organized into columns. The goal is to enter the data into gradient boosting machine algorithm for a specific prediction.

Some columns have more than one entry for the same sample, i.e., sample1 for column1 has entry1 for line1 and entry2 for line2. Here's an example:

df = pd.DataFrame({'pat': [1, 2, 3, 3, 3, 3], 'diag_type': ['D', 'OP', 'D', 'D', 'D', 'OP'],
                  'diag': ['D_1', 'OP_1', 'D_1', 'D_3', 'D_4', 'OP_2']})

    pat     diag_type   diag
0   1   D   D_1
1   2   OP  OP_1
2   3   D   D_1
3   3   D   D_3
4   3   D   D_4
5   3   OP  OP_2

Since I have so many categories I would need to do some restructuring, and I decided that entity embedding is the best way to do it. However, I haven't figure out a good way to restructure the dataframe. I thought about doing some sort of dummy coding as exemplified in this stackoverflow post, e.g.:

    diag_type_D     diag_type_OP    diag_D_1    diag_D_3    diag_D_4    diag_OP_1   diag_OP_2
1   1   0   1   0   0   0   0
2   0   1   0   0   0   1   0
3   3   1   1   1   1   0   1

But then I still get a lot of sparsity and it takes forever. Is there a better way of doing it?


3 Answers 3


Why don't you try with a different kind of encoding?

One of the most typical encodings, when you have high cardinality, is Target Encoding. You can find an implementation in this library. You can find an explanation of what is it and when to apply it here.

  • $\begingroup$ Thanks, I looked into target encoding. My problem is the multiple entries for each ID, so I have more than 1 row for one ID. How do I deal with that? $\endgroup$ Jan 15, 2020 at 15:08

You should only use dummy coding (a.k.a one-hot-encoding) if the cardinality (number of different categories) of each categorical column is quite low. Otherwise, you will end up having too many columns. I would advise you to check the cardinality of each column. If the cardinalities are small (in comparison with the number of rows of your dataset), you can do one-hot-encoding. If the cardinality is large for a column you have to find a different encoding for that column e.g. target statistics.

However, I would advise trying the library catboost. It is a gradient boosting library and naturally handles categorical columns well (it chooses the encoding for you). It is quite easy to use, like this:

from catboost import CatBoostRegressor
from catboost import Pool

train_data = Pool(X_train, y_train, cat_features = categorical_features)
eval_data=  Pool(X_eval, y_eval,  cat_features = categorical_features)
test_data =  Pool(X_test, y_test,  cat_features = categorical_features)

# Initialize CatBoostRegressor
model = CatBoostRegressor()
# Fit model
model.fit(train_data, eval_set = eval_data)
# Get predictions
preds = model.predict(test_data)

Here, categorical_features is a list of column names of all the categorical features of your data. Be sure to include also columns that appear to be numerical, but where neighboring observations do not have a semantic relation. For example, a column containing ID-values has a numerical datatype, but two objects having a similar ID usually does not imply they share some property. Thus this ID column should also be encoded like a categorical column.

There is also a classifier if you do classification.

Regarding your comment about multiple rows having the same category:

Imagine you have a database of fruits. One column holds the names: "Apple", "Banana", "Lemon"... Another column holds the colors: "red", "yellow", "yellow"... The name column would be useless for a model, there is nothing to generalize from it. The name is different for every fruit. The color, however, may be used by a model, because when predicting a property of a fruit not in the training data, it might have the same color as some fruits from the training data. It will never have the same name though.

  • 1
    $\begingroup$ Good idea. However, how do I deal with multiple entries (multiple rows) for one ID? That's the first problem I ran into. $\endgroup$ Jan 15, 2020 at 15:05
  • 1
    $\begingroup$ That's the standard case. If that would not be the case the feature would be meaningless. You can only derive patterns from the data if there are patterns. A column where each entry has its own category is not useful, because when you predict for new instances the model will see new categories it has not seen in the training data. $\endgroup$
    – PascalIv
    Jan 16, 2020 at 8:59
  • $\begingroup$ @wiggalicious I edited the answer with an example $\endgroup$
    – PascalIv
    Jan 16, 2020 at 9:06
  • $\begingroup$ That's a good point. Thanks again for your answer. The problem is more complicated than I anticipated. For new data samples the number of features differ, so it's not really structured in that sense. Maybe sequence to sequence learning is more appropriate here. $\endgroup$ Jan 20, 2020 at 9:35
  • $\begingroup$ Well, you should have included that in the question. $\endgroup$
    – PascalIv
    Jan 20, 2020 at 10:42

You can also use a simple feed forward NN with Embedding layers to tackle these high cardinality features

Use the LabelEncoder or the OrdinalEncoder from sklearn or category-encoders package. You can feed the encoded values directly into a Embedding layer.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.