Categorical variables with multiple entries transformed to entity embedding

Question

I have structured data with lots (tens of thousads) of categories organized into columns. The goal is to enter the data into gradient boosting machine algorithm for a specific prediction.

Some columns have more than one entry for the same sample, i.e., sample1 for column1 has entry1 for line1 and entry2 for line2. Here's an example:

df = pd.DataFrame({'pat': [1, 2, 3, 3, 3, 3], 'diag_type': ['D', 'OP', 'D', 'D', 'D', 'OP'],
                  'diag': ['D_1', 'OP_1', 'D_1', 'D_3', 'D_4', 'OP_2']})

    pat     diag_type   diag
0   1   D   D_1
1   2   OP  OP_1
2   3   D   D_1
3   3   D   D_3
4   3   D   D_4
5   3   OP  OP_2

Since I have so many categories I would need to do some restructuring, and I decided that entity embedding is the best way to do it. However, I haven't figure out a good way to restructure the dataframe. I thought about doing some sort of dummy coding as exemplified in this stackoverflow post, e.g.:

d.get_dummies(df.set_index('pat')).sum(level=0)
    diag_type_D     diag_type_OP    diag_D_1    diag_D_3    diag_D_4    diag_OP_1   diag_OP_2
pat                             
1   1   0   1   0   0   0   0
2   0   1   0   0   0   1   0
3   3   1   1   1   1   0   1

But then I still get a lot of sparsity and it takes forever. Is there a better way of doing it?

Answer 1

Why don't you try with a different kind of encoding?

One of the most typical encodings, when you have high cardinality, is Target Encoding. You can find an implementation in this library. You can find an explanation of what is it and when to apply it here.

Answer 2

You should only use dummy coding (a.k.a one-hot-encoding) if the cardinality (number of different categories) of each categorical column is quite low. Otherwise, you will end up having too many columns. I would advise you to check the cardinality of each column. If the cardinalities are small (in comparison with the number of rows of your dataset), you can do one-hot-encoding. If the cardinality is large for a column you have to find a different encoding for that column e.g. target statistics.

However, I would advise trying the library catboost. It is a gradient boosting library and naturally handles categorical columns well (it chooses the encoding for you). It is quite easy to use, like this:

from catboost import CatBoostRegressor
from catboost import Pool

train_data = Pool(X_train, y_train, cat_features = categorical_features)
eval_data=  Pool(X_eval, y_eval,  cat_features = categorical_features)
test_data =  Pool(X_test, y_test,  cat_features = categorical_features)

# Initialize CatBoostRegressor
model = CatBoostRegressor()
# Fit model
model.fit(train_data, eval_set = eval_data)
# Get predictions
preds = model.predict(test_data)

Here, categorical_features is a list of column names of all the categorical features of your data. Be sure to include also columns that appear to be numerical, but where neighboring observations do not have a semantic relation. For example, a column containing ID-values has a numerical datatype, but two objects having a similar ID usually does not imply they share some property. Thus this ID column should also be encoded like a categorical column.

There is also a classifier if you do classification.

Regarding your comment about multiple rows having the same category:

Imagine you have a database of fruits. One column holds the names: "Apple", "Banana", "Lemon"... Another column holds the colors: "red", "yellow", "yellow"... The name column would be useless for a model, there is nothing to generalize from it. The name is different for every fruit. The color, however, may be used by a model, because when predicting a property of a fruit not in the training data, it might have the same color as some fruits from the training data. It will never have the same name though.

Answer 3

You can also use a simple feed forward NN with Embedding layers to tackle these high cardinality features

Use the LabelEncoder or the OrdinalEncoder from sklearn or category-encoders package. You can feed the encoded values directly into a Embedding layer.