How can I perform categorical encoding when the dataset is too large for memory?

Question

I generally do preprocessing before fitting estimators using Scikit-Learn. My latest project is using significantly more data than I have used in the past, and whilst I know I can use online learning with either Keras using .fit_generator() or sklearn using .partial_fit(), I'm kinda at a loss as to how to do the categorical encoding steps in such a scenario. Obviously OrdinalEncoder requires a knowledge of every possible value in the feature to fit itself fully...but I can't give it that because I can't load the data into memory.

I haven't tried, but it might be possible to load a single categorical feature at a time and train an encoder on that data, before deleting it from memory and loading the next feature to train another encoder. That seems quite clunky though given there's not actually any reason something like categorical encoder couldn't partially learn categories in an iterative fashion, but I can't find anything in sklearn that will accommodate that.

Is there no "online learning" equivalent for preprocessing data?

Answer 1

You can read the file line by line (or block by block), assuming it is in a format where that can be done, and keep track of the unique values of the category.

Can handle multiple fields in 1 pass of the data instead of the 1 field like the below code.

Of course the easiest way is to get more memory or move to a cluster. The line-by-line solution will not be the fastest.

In pseudocode

unique_values_in_field_1 = set()

file = open(input)
while not EOF
    line = readline(file) 
    # can also read a block of lines and adjust the code accordingly
    field_1 = line.field_1
    # make a function to handle multiple fields during the same pass of reading the file
    if field_1 not in unique_values_in_field_1:
       unique_values_in_field_1.add(field_1)

close(file)

# now have all of the unique values in this field. Do whatever is needed to transform them.

Answer 2

I haven't tried, but it might be possible to load a single categorical feature at a time and train an encoder on that data, before deleting it from memory and loading the next feature to train another encoder.

In the end, I just did this.