I have a data frame with around 37,000 rows and 54 columns. Out of these 54 columns, two columns namely 'user_id' and 'mail_id' are provided in a very creepy format as shown below:
After a detailed analysis of my data, I figured out that I cannot drop these two columns from my data frame as they are too important for prediction. I can hash these two features but there is one more interesting thing. There are only 2,000 types of user_ids and mail_ids. So doing one hot encoding can help a lot. My question is that if I convert this into one hot encoding using 'get_dummies' method in pandas with
sparse = True, will it be memory efficient or is there any other efficient way to do it?