I generally do preprocessing before fitting estimators using Scikit-Learn. My latest project is using significantly more data than I have used in the past, and whilst I know I can use online learning with either Keras using .fit_generator()
or sklearn using .partial_fit()
, I'm kinda at a loss as to how to do the categorical encoding steps in such a scenario. Obviously OrdinalEncoder requires a knowledge of every possible value in the feature to fit itself fully...but I can't give it that because I can't load the data into memory.
I haven't tried, but it might be possible to load a single categorical feature at a time and train an encoder on that data, before deleting it from memory and loading the next feature to train another encoder. That seems quite clunky though given there's not actually any reason something like categorical encoder couldn't partially learn categories in an iterative fashion, but I can't find anything in sklearn that will accommodate that.
Is there no "online learning" equivalent for preprocessing data?