Whenever we have a dataset to be pre processed, before feeding it to the model we convert the categorical values to numerical values for which we generally use LabelEncoding, One Hot encoding etc techniques but all these are done manually going through each column.

But what if are dataset is huge in terms of columns(eg : 2000 columns), here it wont be possible to go through each column manually, in such cases how do we handle encoding?

Are there any specific libraries available which deal with automatic encoding of variable? I know of category_encoders which provides with different encoding techniques but how do we do it at in the above mentioned condition.

  • $\begingroup$ What language. In R - for instance - you can use model matrix to encode features in bulk github.com/Bixi81/R-ml/blob/master/prep_factor_to_dummies.R $\endgroup$ – Peter Nov 5 '20 at 8:39
  • $\begingroup$ that is a really a good to know information, I would look into it, thanks! I wanted to know if there is something in Python. $\endgroup$ – Sahil Nov 5 '20 at 10:07
  • $\begingroup$ stackoverflow.com/questions/10196860/… $\endgroup$ – Peter Nov 5 '20 at 11:21
  • $\begingroup$ What is your ask. How can we encode 2K columns? category_encoders will do that. Or how to decide when to use OHE Or Label etc for 2K features? $\endgroup$ – 10xAI Nov 5 '20 at 15:12
  • $\begingroup$ "how to decide when to use OHE Or Label etc for 2K features?" $\endgroup$ – Sahil Nov 6 '20 at 16:09

To correctly encode your variable you have to know what those variables are about. An algorithm needs to somehow understand the type of your variable to automatically encode them. In such a case, you should have a dictionary for variables about the type of variable (sometimes, dataset guides or readme files, or some txt files contain that). Or you have to know that all the variables are monotype, so you can apply the same encoding. If you don't have these or similar information sources about the data, it is not possible (unless you have a perfect model to categorize the variables according to their type :)) ) to automatically encode them.

Although you cannot categorize the types of categorical variables automatically, it is possible to distinguish the continuous (and also discrete) and categorical variables. When I face a similar situation of having lots of variables, one of the very first things that I do is to have a count and percentage of distinct values for every variable. Thus, for example, if a variable with 200000 samples has ~154000 (unless there is a variable with 154000 categories, which is almost possible) distinct values, then it is a continuous (or discrete) variable. If a variable with 200000 samples has 13 distinct values then it is a categorical variable for sure. Using similar tricks you can identify categorical variables. However, thereafter, it is inevitable to analyze categorical variables one-by-one. When you categorize them within themselves e.g. rank variables, nominal variables, etc. you can altogether encode each variable type.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.