# Handling encoding of a dataset which has more than total 2000 columns

Whenever we have a dataset to be pre processed, before feeding it to the model we convert the categorical values to numerical values for which we generally use LabelEncoding, One Hot encoding etc techniques but all these are done manually going through each column.

But what if are dataset is huge in terms of columns(eg : 2000 columns), here it wont be possible to go through each column manually, in such cases how do we handle encoding?

Are there any specific libraries available which deal with automatic encoding of variable? I know of category_encoders which provides with different encoding techniques but how do we do it at in the above mentioned condition.

• What language. In R - for instance - you can use model matrix to encode features in bulk github.com/Bixi81/R-ml/blob/master/prep_factor_to_dummies.R Nov 5, 2020 at 8:39
• that is a really a good to know information, I would look into it, thanks! I wanted to know if there is something in Python. Nov 5, 2020 at 10:07
• stackoverflow.com/questions/10196860/… Nov 5, 2020 at 11:21
• What is your ask. How can we encode 2K columns? category_encoders will do that. Or how to decide when to use OHE Or Label etc for 2K features? Nov 5, 2020 at 15:12
• "how to decide when to use OHE Or Label etc for 2K features?" Nov 6, 2020 at 16:09

Now once you know what kind of variables you have, you can directly apply the relevant techniques to those column only by using the library you mentioned category_encoders. Suppose you have 5 columns which need One Hot Encoding, you do not apply One Hot Encoding to all 5 of them separately. Just mentions the column names when applying the encoder and it will apply automatically.