0
$\begingroup$

I have 2 features productName and productCategory , both of them are strings. I have a category named supplier. There are 4000 suppliers and 500,000 items in test data. I don't think one hot encoding will be a good approach to deal with such a big data. How should i handle these using random forest?

Improve this question
$\endgroup$
2
  • $\begingroup$ So, to sum up, how many features you will have if you do OneHotEncoder? And how many samples? $\endgroup$
    – Alex Serra Marrugat
    Commented Apr 6, 2021 at 7:57
  • $\begingroup$ Features are only 2 but sample size is 500,000. One-hot encoder will take too much time $\endgroup$
    – anita shrivastava
    Commented Apr 6, 2021 at 8:35

1 Answer 1

Reset to default
0
$\begingroup$

You have a few options on your hand:

Option 1 - Augment you dataset with additional Supplier-related features Do you have additional features about the supplier that makes sense to augment into your dataset ? For e.g. you might be able to replace the Supplier field with something like "Years in Business" , "Primary Industry", "Reputation Score" etc. That way you convert the categorical variable into something more meaningful that can be compared across the universe of Suppliers.

Go this route if and only if you have meaningful and easily accessible features for your problem domain.

Option 2 - Encode each Supplier with a unique integer I know this sounds stupid - but it really works for Random Forest models (wouldn't work on simple Linear Regression for sure). Random Forest models are scale invariant. They won't get biased towards suppliers with a larger numeric Supplier Id. They'll figure out the best way to split your Supplier Ids to meet your objective function.

I would recommend Option 2 to start with since that is the simplest.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Can i do it with features also? Features are also strings and the sample size is 500,000. Hot-encoding will make it much complex. $\endgroup$
    – anita shrivastava
    Commented Apr 6, 2021 at 8:36
  • $\begingroup$ yes, you can follow the same for all features - for Random Forest models $\endgroup$
    – Jayaram Iyer
    Commented Apr 6, 2021 at 16:00
  • 1
    $\begingroup$ It will work for rf in the sense that they will be able to build trees. But for sure those trees won’t be optimal since you induce an ordering, otherwise not existing in data $\endgroup$
    – rapaio
    Commented Apr 7, 2021 at 17:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.