1
$\begingroup$

I'm using an older version of SciKit-Learn, version 1.0.2, to try and OneHotEncode some data. My dataset is fairly large, 184 columns going to 311 after the OneHotEncoding, with ~500,000 rows. Despite this, I'm confident I could write code that OneHotEncodes my columns in a minute maximum.

Currently, SciKit-Learn's OneHotEncoder is on 10 minutes and counting. Why is this code so slow? Is there anything I can do to speed it all up?

$\endgroup$

1 Answer 1

3
$\begingroup$

If you use OneHotEncoder in a Jupyter Notebook, you can use %%prun -s "time" to profile your code. See How do I Profile a Jupyter Cell and Rank it by Cumulative Time? for more information.

Doing this with a subset of the rows shows that the function calls taking up the most time are inside the _encode module function from SciKit-Learn's utils sub-package.

enter image description here

Looking inside this module you can see that _encode's _check_unknown, and _extract_missing contain inefficient list comprehensions, and this is most likely what's slowing you down.

If you want to create a OneHotEncoding of your data I suggest using Panda's get_dummies functionality, it's so much faster.

There are a couple of differences, pd.get_dummies(...) disregards any NaN values by default, this can be changed by switching dummy_na=True. It is also sometimes considered best practice to drop one of the categorical features as a column, as one of the columns can be represented by all zeros in the other columns. For more information on that try reading these articles: dummy variable trap, does the dropped column matter?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.