0
$\begingroup$

I have a huge data set with one of the columns named 'mail_id'. The mail_id is given in a very creepy format as shown below:

mail_id
DQ/4I+GIOz2ZoIiK0Lg0AkwnI35XotghgUK/MYc101I=
BL3z4RtiyfIDydaRYWX2ZXL6IX10QH1yG5ak1s/8Lls=
BL3z4RtiyfIDydaRYWX2ZXL6IX10QH1yG5ak1s/8Lls=
EHNBRbi6i9KO6cMHsuDPFjZVp2cY3RH+BiOKwPwzLQs=
K0y/NW59TJkYc5y0HUwDeAXrewYT0JQlkcozz0s2V5Q=
UGATDXARg7jMEInKH7oXgty2nwxnwD2l0OW/8Nsa0MI=
qE9zgWiITYA97RUiN4X/t9IVWLViLz+lKijaYegyBiQ=
BL3z4RtiyfIDydaRYWX2ZXL6IX10QH1yG5ak1s/8Lls=
4+EEK8RbNYwuFCHznY9XSRCV4Yek60bHVgnP3jtjjzk=

After doing a lot of analysis on my data, I have found that I cannot drop this feature set from my model so I have to convert it to something meaningful. Can anyone please explain me how to do this efficiently?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ It just appears to be hashed for privacy. There's probably no reason you'd want to throw away this feature -- just use it as a factor. After all, you can see right off the bat that some of the ID's appear repeatedly, so this is probably an extremely useful feature as it gives you a way to identify which rows correspond to the same individuals. $\endgroup$
    – Hack-R
    Aug 30, 2016 at 15:19
  • $\begingroup$ Exactly @Hack-R but in order to make my model memory efficient I have to convert it into something else. How can I do that ? $\endgroup$
    – enterML
    Aug 30, 2016 at 15:26
  • $\begingroup$ Hashes are usually the most memory efficient. You can't use that? If not then I would just assign a sequence of integer or character ID's. $\endgroup$
    – Hack-R
    Aug 30, 2016 at 15:34

1 Answer 1

Reset to default
2
$\begingroup$

I'd say there are pros and cons of using FeatureHasher for this purpose. If you really striving to use it, then just instantiate it like this:

In [1]:
from sklearn.feature_extraction import FeatureHasher
h = FeatureHasher(n_features=5, input_type='string')
f = h.transform(mail_id)
f.toarray()

Out[1]:
array([[ 1.,  0.,  0.,  0.,  0.],
      [ 0., -1.,  0.,  0.,  0.],
      [ 1.,  0.,  0.,  0.,  0.],
      [ 0.,  0., -1.,  0.,  0.],
      [ 0., -1.,  0.,  0.,  0.]])

So, after you have instantiated it, just .transform each your upcoming mail_id and use results in upstream applications ( like online learning, for instance ). Obviously n_features is some knob to tune. But this has its flip side: the cardinality of mail ids is apriori high, so unless you have very limited amount of users you will need enormous n_features to minimize collisions.

The better would be to take logs, where your ids coappear, and learn item2vec style model. This will deliver much denser (and meaningful) representation of mail_ids than FeatureHasher would do.

Also, take a look at this.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.