How to predict on data that is label encoded as end user will input a categorical data?

Question

My dataset contains about 29 features with 3 class labels as result. Among these 29 features around 24 features are categorical i cannot transform each category into numbers as there are many more than 30+ categories in some features.

What i did is label Encoding.As my whole dataframe is encoded with label encoder how would i do prediction on it as i dont know what numbers have been used by the labelencoder for the data. Also what if a end user inputs a category which is not already present in the dataframe and has not been encoded earlier?

from sklearn import preprocessing
le = preprocessing.LabelEncoder()

data=data.apply(le.fit_transform)

This code has encoded my dataframe into numbers Now what should i do if i want to either print a particular category in its numeric transformed digit? Should i use label encoder or any other technique as i have applied Random Forest on my dataset.

Answer 1

This approach is unlikely to work well imho. I think you should analyze your data more thoroughly before applying technical workarounds to the features.

• First, the label encoder represents values as integer. This introduces an order which can cause bias in the model. In particular there is a higher risk of overfitting.
• If you do this, clearly you must store the correspondence between the category and the integer. You will need to be able to convert in both directions.
• A new category doesn't make sense with a supervised model: the model has not been trained on it so what can it do with it?

A lot depends on the size of the dataset (what is it?), but in my opinion a better method would be the following:

• First, measure the correlation of every feature with the target. If the correlation is close to zero, the feature can be removed.
• Each categorical feature left should preferably be one hot encoded. But rare values should be removed first (for example if $n\leq 3$), because these cases would cause overfitting and are unlikely to contribute to knowing the target. - A common way to remove rare values for a feature do this is to assign a special unknown value instead of the original value. This has the advantage that it can also be used when applying the model to any feature value which was not known at the time of training.

This process should strongly reduce the number of values, hence make the final number of features more reasonable.

Answer 2

For high number of categorical variables, I recommend using CatBoost since that library is optimized for such data. Although, I do not see an issue using RandomForestClassifier for Multi-Class labeling (3 classes).

For handling category encoding and input categories, I recommend using OrdinalEncoder for features. LabelEncoder should be used for features with known categories like hot/cold/unkown and target label where classes are already known. Simply save the Encoder object in the model directory and load it everytime a new input needs to be evaluated and prediction needs to be made. There is an option to pass an argument 'unknown_value' which the new unseen categories will be assigned. This way the model will use only the information presented during training to evaluate and new categories will not make a difference. And you can always choose to update the encoder for new features when training again.