I have a huge dataset with categorical data. It is comprised of alerts having multiple properties. Each alert belongs to a group, and some even belong to multiple groups. It looks somewhat like this:

     GroupID           System        State       TimeStamp        etc...
0    [1, 2, 3, 4]         A           REC           ...
1    [1, 2, 3, 4]         A           SNT           ...
2    [2, 4]               B           REC 
3    [2, 4]               B           PND
4    [2, 4]               B           COM
5    [2, 4]               B           SNT
6    [2]                  C           RCV
7    [2]                  C           ACC

There are more than 100000 different group IDs in over 3 mil alerts.

  1. Creating a column with a single Group ID value (not a list) means some alerts will appear more than once, which is not good given the already huge dataset.
  2. Creating a separate column for each group (binary encoding) would expand my data too much horizontally.

What is a memory efficient way of encoding Groups?

  • $\begingroup$ How many unique combinations of multiple groups, and do you need to keep the detail of every group id in a combination? If not you could consider replacing every combination with a single new id. $\endgroup$
    – Erwan
    Commented Feb 14, 2020 at 17:27
  • $\begingroup$ @Erwan The number of combinations would be even greater than the number of group IDs... $\endgroup$
    – redguy
    Commented Feb 18, 2020 at 15:07

1 Answer 1


There is several techniques that could work for you:


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.