If you use OneHotEncoder
in a Jupyter Notebook, you can use %%prun -s "time"
to profile your code. See How do I Profile a Jupyter Cell and Rank it by Cumulative Time? for more information.
Doing this with a subset of the rows shows that the function calls taking up the most time are inside the _encode
module function from SciKit-Learn's utils
sub-package.

Looking inside this module you can see that _encode
's _check_unknown
, and _extract_missing
contain inefficient list comprehensions, and this is most likely what's slowing you down.
If you want to create a OneHotEncoding
of your data I suggest using Panda's get_dummies
functionality, it's so much faster.
There are a couple of differences, pd.get_dummies(...)
disregards any NaN values by default, this can be changed by switching dummy_na=True
. It is also sometimes considered best practice to drop one of the categorical features as a column, as one of the columns can be represented by all zeros in the other columns. For more information on that try reading these articles: dummy variable trap, does the dropped column matter?