1
$\begingroup$

I have multiple log records with discrete categorical features. Shape of my dataset is (100k, 24) My aim is to look for anomalies in these records. I am planning to cluster the data after encoding.

Before going forward with any analysis, the categorical features need to be encoded; I'd have ideally gone with LabelEncoder or OneHotEncoder but the issue is some features have > 40 possible values.

As of now, I've frequency encoded the variables but is there something I am missing and is there a better way of going about this?

| improve this question | |
$\endgroup$
1
$\begingroup$

High cardinality is a common issue that we run into when doing encoding for categorical variables. One hot encoding and label encoding are the most common approaches however label encoding maynot be ideal if the variable is not nominal (not inherently hierarchical) and one hot encoding will lead to curse of dimensionality when no of categories are high. REplacing with ratio or looking at spectral encoding is a good approach to circumvent this

Details of spectral encoding are :- https://towardsdatascience.com/spectral-encoding-of-categorical-features-b4faebdf4a

| improve this answer | |
$\endgroup$
-1
$\begingroup$

you need to use OneHotEncoder() for you independent variables (X), and LabelEncoder() for your dependent variable (y).

If you want to reduce the number of dummy variables (output of OneHotEncoder) you can remove one dummy variable for each category. For example, for a categorical variable, Gender, you will have Gender_Female and Gender_Male; but only one is required. Gender_Female=0 means Male. So you need to use drop='first' for OneHotEncoder(). However, when you use this feature, you have to use handle_unknown='error' (the latest version has a limitation than you can't use handle_unknown='ignore'). I highly recommend you to read this article Choosing the right Encoding method-Label vs OneHot Encoder.

The other possible way to reduce the number of variables, but I don't recommend', is using feature selection methods, SelectKBest(). However, I don't believe it works for non-linear problems as I haven't got good results in my problems. You can read here more 1.13. Feature selection

A better solution may be dimensionality reduction methods, e.g. PCA, to reduce the number of apparent variables and make your modelling computationally lighter and faster; but you know that you may lose some accuracy when you use this method. Read here 6.5. Unsupervised dimensionality reduction

| improve this answer | |
$\endgroup$
  • $\begingroup$ Hi, there is no dependent variable. That's why I was thinking of an unsupervised learning method. And OneHotEncoding would just increase the dimension of the dataset to a huge number, some categorical features have >40 unique values $\endgroup$ – Raghav Kukreti Dec 12 '19 at 5:00
  • $\begingroup$ If there is no Y, you should definitely go for an unsupervised methods. SelectKBest() needs a dependet variable; but your best choice might be PCA (dimensionality reduction methods). You can also drop dummy variables with a lot of zero (few 1, for instance less than 5%) $\endgroup$ – Mehdi Dec 12 '19 at 7:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.