I have a data frame with around 37,000 rows and 54 columns. Out of these 54 columns, two columns namely 'user_id' and 'mail_id' are provided in a very creepy format as shown below:

user_id                                           mail_id       
AR+tMy3H/E+Re8Id20zUIz+amJkv6KU12o+BrgIDin0=      DQ/4I+GIOz2ZoIiK0Lg0AkwnI35XotghgUK/MYc101I=
1P4AOvdzJzhDSHi7jJ3udWv4ajpKxOn4T/rCLv4PrXU=      BL3z4RtiyfIDydaRYWX2ZXL6IX10QH1yG5ak1s/8Lls=
OEfFUcsTAGInCfsHuLZuIgdSNtuNsg8EdfN98VUZVTs=      BL3z4RtiyfIDydaRYWX2ZXL6IX10QH1yG5ak1s/8Lls=   
1P4AOvdzJzhDSHi7jJ3udWv4ajpKxOn4T/rCLv4PrXU=      EHNBRbi6i9KO6cMHsuDPFjZVp2cY3RH+BiOKwPwzLQs=
CYRcuV0cR0algMZJ1N6+3uKcqi8iu+6tJNzmBbmgN7o=      K0y/NW59TJkYc5y0HUwDeAXrewYT0JQlkcozz0s2V5Q=

After a detailed analysis of my data, I figured out that I cannot drop these two columns from my data frame as they are too important for prediction. I can hash these two features but there is one more interesting thing. There are only 2,000 types of user_ids and mail_ids. So doing one hot encoding can help a lot. My question is that if I convert this into one hot encoding using 'get_dummies' method in pandas with sparse = True, will it be memory efficient or is there any other efficient way to do it?

  • 1
    $\begingroup$ Just do it; that's not a lot of data even for a laptop. $\endgroup$ – Emre Sep 1 '16 at 2:32

@Nain, this is called a hard problem :)

One possible solution is called "Engineering Mixed Variables." I've attached screenshot from a possible solution. Not everything out of fairness to the author. It's based on Soledad Galli's Feature Engineering for Machine Learning on Udemy.

enter image description here


@Student T: Pandas can have memory issues with only a few thousand rows, depends on width and operation (merge() can be bad).

If you do have issues I've found moving to a database the least painful, just dump your dataframes to tables with DataFrame.to_sql() and pull it back into your scikit-learn pipeline with pandas.read_sql_query()


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.