I have a huge data set with one of the columns named 'mail_id'. The mail_id is given in a very creepy format as shown below:


After doing a lot of analysis on my data I have found that I cannot drop this feature set from my model so I have to convert it to something meaningful. Can anyone please explain me how to do this efficiently? Thanks in advance.

  • 1
    $\begingroup$ It just appears to be hashed for privacy. There's probably no reason you'd want to throw away this feature -- just use it as a factor. After all, you can see right off the bat that some of the ID's appear repeatedly, so this is probably an extremely useful feature as it gives you a way to identify which rows correspond to the same individuals. $\endgroup$ – Hack-R Aug 30 '16 at 15:19
  • $\begingroup$ Exactly @Hack-R but in order to make my model memory efficient I have to convert it into something else. How can I do that ? $\endgroup$ – enterML Aug 30 '16 at 15:26
  • $\begingroup$ Hashes are usually the most memory efficient. You can't use that? If not then I would just assign a sequence of integer or character ID's. $\endgroup$ – Hack-R Aug 30 '16 at 15:34

I'd say there are pros and cons of using FeatureHasher for this purpose. If you really striving to use it, then just instantiate it like this:

In [1]:
from sklearn.feature_extraction import FeatureHasher
h = FeatureHasher(n_features=5, input_type='string')
f = h.transform(mail_id)

array([[ 1.,  0.,  0.,  0.,  0.],
      [ 0., -1.,  0.,  0.,  0.],
      [ 1.,  0.,  0.,  0.,  0.],
      [ 0.,  0., -1.,  0.,  0.],
      [ 0., -1.,  0.,  0.,  0.]])

So, after you have instantiated it, just .transform each your upcoming mail_id and use results in upstream applications ( like online learning, for instance ). Obviously n_features is some knob to tune. But this has its flip side: the cardinality of mail ids is apriori high, so unless you have very limited amount of users you will need enormous n_features to minimize collisions.

The better would be to take logs, where your ids coappear, and learn item2vec style model ( https://arxiv.org/pdf/1603.04259v2.pdf ). This will deliver much denser ( and meaningful ) representation of mail_ids than FeatureHasher would do.

Also take a look at this: http://www.win-vector.com/blog/2012/07/modeling-trick-impact-coding-of-categorical-variables-with-many-levels/

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.