# Memory efficient encoding logic for group categories

I have a huge dataset with categorical data. It is comprised of alerts having multiple properties. Each alert belongs to a group, and some even belong to multiple groups. It looks somewhat like this:

     GroupID           System        State       TimeStamp        etc...
0    [1, 2, 3, 4]         A           REC           ...
1    [1, 2, 3, 4]         A           SNT           ...
2    [2, 4]               B           REC
3    [2, 4]               B           PND
4    [2, 4]               B           COM
5    [2, 4]               B           SNT
6    [2]                  C           RCV
7    [2]                  C           ACC
...


There are more than 100000 different group IDs in over 3 mil alerts.

1. Creating a column with a single Group ID value (not a list) means some alerts will appear more than once, which is not good given the already huge dataset.
2. Creating a separate column for each group (binary encoding) would expand my data too much horizontally.

What is a memory efficient way of encoding Groups?

• How many unique combinations of multiple groups, and do you need to keep the detail of every group id in a combination? If not you could consider replacing every combination with a single new id. Feb 14, 2020 at 17:27
• @Erwan The number of combinations would be even greater than the number of group IDs... Feb 18, 2020 at 15:07