1
$\begingroup$

I have dataset like :

   profile     category  target
0        1      [5, 10]       1
1        2          [1]       0
2        3   [23, 5000]       1
3        4  [700, 4500]       0

How to handle category feature, this table may have others additional features too. One hot encoding lead to consume too much space.because number of rows is around 10 million. Any suggestion would be helpful.

$\endgroup$
3
$\begingroup$

I recommend using scikit-learn's MultiLabelBinarizer to encode your category feature. This is the multi-label analog of one-hot encoding, but it has a sparse_output mode, which should solve your problem of limited memory.

$\endgroup$
  • $\begingroup$ This dataset also have other numerical and categorical features. how can i combine those sparse and non sparse feature together for machine learning modeling. $\endgroup$ – GIRISH kuniyal Nov 21 at 4:47
  • 1
    $\begingroup$ I solved sparse vs non sparse dataset combining problem by making non-sparse feature to sparse then combine both sparse feature matrix using scipy.sparse.hstack() $\endgroup$ – GIRISH kuniyal Nov 22 at 7:20
0
$\begingroup$

MultiLabelBinarizer is an good choice. For sparse features i would say first do some feature engineering on them. If necessary then keep them else discard them. If you keep them then by applying error reduction methods helps a lot. They will fill the sparse features & then by encoding the categorical features you can apply machine learning models on them.

I was stuck on features with more sparse values so i followed the above approach. Worked fine with me.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.