1
$\begingroup$

I have dataset like :

   profile     category  target
0        1      [5, 10]       1
1        2          [1]       0
2        3   [23, 5000]       1
3        4  [700, 4500]       0

How to handle category feature, this table may have others additional features too. One hot encoding lead to consume too much space.because number of rows is around 10 million. Any suggestion would be helpful.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
$\begingroup$

I recommend using scikit-learn's MultiLabelBinarizer to encode your category feature. This is the multi-label analog of one-hot encoding, but it has a sparse_output mode, which should solve your problem of limited memory.

Improve this answer
$\endgroup$
2
  • $\begingroup$ This dataset also have other numerical and categorical features. how can i combine those sparse and non sparse feature together for machine learning modeling. $\endgroup$
    – GIRISH kuniyal
    Nov 21, 2019 at 4:47
  • 1
    $\begingroup$ I solved sparse vs non sparse dataset combining problem by making non-sparse feature to sparse then combine both sparse feature matrix using scipy.sparse.hstack() $\endgroup$
    – GIRISH kuniyal
    Nov 22, 2019 at 7:20
0
$\begingroup$

MultiLabelBinarizer is an good choice. For sparse features i would say first do some feature engineering on them. If necessary then keep them else discard them. If you keep them then by applying error reduction methods helps a lot. They will fill the sparse features & then by encoding the categorical features you can apply machine learning models on them.

I was stuck on features with more sparse values so i followed the above approach. Worked fine with me.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.