I have structured data with lots (tens of thousads) of categories organized into columns. The goal is to enter the data into gradient boosting machine algorithm for a specific prediction.
Some columns have more than one entry for the same sample, i.e., sample1 for column1 has entry1 for line1 and entry2 for line2. Here's an example:
df = pd.DataFrame({'pat': [1, 2, 3, 3, 3, 3], 'diag_type': ['D', 'OP', 'D', 'D', 'D', 'OP'],
'diag': ['D_1', 'OP_1', 'D_1', 'D_3', 'D_4', 'OP_2']})
pat diag_type diag
0 1 D D_1
1 2 OP OP_1
2 3 D D_1
3 3 D D_3
4 3 D D_4
5 3 OP OP_2
Since I have so many categories I would need to do some restructuring, and I decided that entity embedding is the best way to do it. However, I haven't figure out a good way to restructure the dataframe. I thought about doing some sort of dummy coding as exemplified in this stackoverflow post, e.g.:
d.get_dummies(df.set_index('pat')).sum(level=0)
diag_type_D diag_type_OP diag_D_1 diag_D_3 diag_D_4 diag_OP_1 diag_OP_2
pat
1 1 0 1 0 0 0 0
2 0 1 0 0 0 1 0
3 3 1 1 1 1 0 1
But then I still get a lot of sparsity and it takes forever. Is there a better way of doing it?