Using scikit-learn FeatureHasher

I have a huge data set with one of the columns named 'mail_id'. The mail_id is given in a very creepy format as shown below:

mail_id
DQ/4I+GIOz2ZoIiK0Lg0AkwnI35XotghgUK/MYc101I=
BL3z4RtiyfIDydaRYWX2ZXL6IX10QH1yG5ak1s/8Lls=
BL3z4RtiyfIDydaRYWX2ZXL6IX10QH1yG5ak1s/8Lls=
EHNBRbi6i9KO6cMHsuDPFjZVp2cY3RH+BiOKwPwzLQs=
K0y/NW59TJkYc5y0HUwDeAXrewYT0JQlkcozz0s2V5Q=
UGATDXARg7jMEInKH7oXgty2nwxnwD2l0OW/8Nsa0MI=
qE9zgWiITYA97RUiN4X/t9IVWLViLz+lKijaYegyBiQ=
BL3z4RtiyfIDydaRYWX2ZXL6IX10QH1yG5ak1s/8Lls=
4+EEK8RbNYwuFCHznY9XSRCV4Yek60bHVgnP3jtjjzk=


After doing a lot of analysis on my data I have found that I cannot drop this feature set from my model so I have to convert it to something meaningful. Can anyone please explain me how to do this efficiently? Thanks in advance.

• It just appears to be hashed for privacy. There's probably no reason you'd want to throw away this feature -- just use it as a factor. After all, you can see right off the bat that some of the ID's appear repeatedly, so this is probably an extremely useful feature as it gives you a way to identify which rows correspond to the same individuals. – Hack-R Aug 30 '16 at 15:19
• Exactly @Hack-R but in order to make my model memory efficient I have to convert it into something else. How can I do that ? – enterML Aug 30 '16 at 15:26
• Hashes are usually the most memory efficient. You can't use that? If not then I would just assign a sequence of integer or character ID's. – Hack-R Aug 30 '16 at 15:34

I'd say there are pros and cons of using FeatureHasher for this purpose. If you really striving to use it, then just instantiate it like this:

In [1]:
from sklearn.feature_extraction import FeatureHasher
h = FeatureHasher(n_features=5, input_type='string')
f = h.transform(mail_id)
f.toarray()

Out[1]:
array([[ 1.,  0.,  0.,  0.,  0.],
[ 0., -1.,  0.,  0.,  0.],
[ 1.,  0.,  0.,  0.,  0.],
[ 0.,  0., -1.,  0.,  0.],
[ 0., -1.,  0.,  0.,  0.]])


So, after you have instantiated it, just .transform each your upcoming mail_id and use results in upstream applications ( like online learning, for instance ). Obviously n_features is some knob to tune. But this has its flip side: the cardinality of mail ids is apriori high, so unless you have very limited amount of users you will need enormous n_features to minimize collisions.

The better would be to take logs, where your ids coappear, and learn item2vec style model ( https://arxiv.org/pdf/1603.04259v2.pdf ). This will deliver much denser ( and meaningful ) representation of mail_ids than FeatureHasher would do.