1
$\begingroup$

My dataset contains about 29 features with 3 class labels as result. Among these 29 features around 24 features are categorical i cannot transform each category into numbers as there are many more than 30+ categories in some features.

What i did is label Encoding.As my whole dataframe is encoded with label encoder how would i do prediction on it as i dont know what numbers have been used by the labelencoder for the data. Also what if a end user inputs a category which is not already present in the dataframe and has not been encoded earlier?

from sklearn import preprocessing
le = preprocessing.LabelEncoder()

data=data.apply(le.fit_transform)

This code has encoded my dataframe into numbers Now what should i do if i want to either print a particular category in its numeric transformed digit? Should i use label encoder or any other technique as i have applied Random Forest on my dataset.

$\endgroup$

1 Answer 1

0
$\begingroup$

This approach is unlikely to work well imho. I think you should analyze your data more thoroughly before applying technical workarounds to the features.

  • First, the label encoder represents values as integer. This introduces an order which can cause bias in the model. In particular there is a higher risk of overfitting.
  • If you do this, clearly you must store the correspondence between the category and the integer. You will need to be able to convert in both directions.
  • A new category doesn't make sense with a supervised model: the model has not been trained on it so what can it do with it?

A lot depends on the size of the dataset (what is it?), but in my opinion a better method would be the following:

  • First, measure the correlation of every feature with the target. If the correlation is close to zero, the feature can be removed.
  • Each categorical feature left should preferably be one hot encoded. But rare values should be removed first (for example if $n\leq 3$), because these cases would cause overfitting and are unlikely to contribute to knowing the target. - A common way to remove rare values for a feature do this is to assign a special unknown value instead of the original value. This has the advantage that it can also be used when applying the model to any feature value which was not known at the time of training.

This process should strongly reduce the number of values, hence make the final number of features more reasonable.

$\endgroup$
4
  • $\begingroup$ Thanks it really helped $\endgroup$ Commented Oct 31, 2022 at 10:55
  • $\begingroup$ As i wanted to do it with the label encoder i have stored the changed labs in an array(not the best way should have used some other data structure) and created a function that inverse those numbers to original categories.Although label encoder has a function of inverse_transform but as whole dataset was labelencoded le.classes_ only has classes for last column so inverse_transform could not work to convert all labels. There are also similar categories in different columns thats why each category with the column name has been stored into an array. $\endgroup$ Commented Oct 31, 2022 at 11:01
  • $\begingroup$ @MuhammadMinhas it's not very clear but apparently you tried to use a single array for all the features, right? If so you must use a different array for each feature, for example by using a dict containing all the arrays as values. It's because labelencoder simply replaces a feature values with integers 1..N, and there's no link between the same integer for two features. $\endgroup$
    – Erwan
    Commented Oct 31, 2022 at 11:10
  • $\begingroup$ I actually used three arrays one to store the features, second to store labels and the third to store the keys or numbers related to that category.Function contains two parameters for header and label and it returns and index and the array containing keys have that particular key for that label under particular header at that index. I believe its not a good approach as it takes a lot of time and space to find out the number for particular category. I open to suggestions what should i do in this case.Although it works fine but i am finding a better option as i am a beginner in python and ML. $\endgroup$ Commented Oct 31, 2022 at 12:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.