Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I'm using an older version of SciKit-Learn, version 1.0.2, to try and OneHotEncode some data. My dataset is fairly large, 184 columns going to 311 after the OneHotEncoding, with ~500,000 rows. Despite this, I'm confident I could write code that OneHotEncodes my columns in a minute maximum.
Currently, SciKit-Learn's OneHotEncoder is on 10 minutes and counting. Why is this code so slow? Is there anything I can do to speed it all up?
If you use OneHotEncoder in a Jupyter Notebook, you can use %%prun -s "time" to profile your code. See How do I Profile a Jupyter Cell and Rank it by Cumulative Time? for more information.
Doing this with a subset of the rows shows that the function calls taking up the most time are inside the _encode module function from SciKit-Learn's utils sub-package.
Looking inside this module you can see that _encode's _check_unknown, and _extract_missing contain inefficient list comprehensions, and this is most likely what's slowing you down.
If you want to create a OneHotEncoding of your data I suggest using Panda's get_dummies functionality, it's so much faster.
There are a couple of differences, pd.get_dummies(...) disregards any NaN values by default, this can be changed by switching dummy_na=True. It is also sometimes considered best practice to drop one of the categorical features as a column, as one of the columns can be represented by all zeros in the other columns. For more information on that try reading these articles: dummy variable trap, does the dropped column matter?
