2
$\begingroup$

I'm working on a short data science project to compare the accuracy of different classification methods. The groups decided to use and compare Random Forest, Naive Bayes and SVM.

The dataset we are using has four categorical features. Each of which has a large number of unique values.

  • There are 16537 unique combinations of 17370 unique values in FeaureA.
  • There are 13860 unique combinations of 13852 unique values in FeaureB.
  • There are 3295 unique combinations of 29 unique values in FeaureC.
  • There are 1518 unique combinations of 29 unique values in FeaureD.

From what I've read the RF and NB algorithms should work fine with label encoding but SVM requires one hot encoding. However that would increase the number of features by ~35K. The performance cost seems like it would be significant. Ideally we would use the same encoding for all three algorithms. Would it be better to take the performance hit and try something like PCA for feature reduction?

Improve this question
$\endgroup$
4
  • $\begingroup$ Depends on the model. It is fine for RF but will not work for SVM, LR etc. $\endgroup$
    – 10xAI
    Commented Apr 16, 2021 at 17:18
  • $\begingroup$ maybe frequency encoding, be better, towardsdatascience.com/… $\endgroup$
    – Nikos M.
    Commented Apr 17, 2021 at 8:29
  • 1
    $\begingroup$ How many instances do you have? You would need a very large dataset to avoid overfitting with this level of diversity in the features. Also what do you mean by "X unique combinations of Y unique values"? It looks like each feature is itself a combination of several things, isn't it? Be aware that any feature value which occurs only once in the data is unusable. $\endgroup$
    – Erwan
    Commented Apr 17, 2021 at 19:05
  • $\begingroup$ The features are things like genre so you have one entry thats action, one thats action;indie, etc Feature A is publisher so nearly all the values are unique I suppose it would be best to eliminate that feature entirely then as it has little to no predictive value? $\endgroup$
    – Jim Jones
    Commented Apr 18, 2021 at 13:41

2 Answers 2

Reset to default
1
$\begingroup$

These are some of the things you could try to reduce the dimensionality of your dataset:-

1.) The first thing to do is feature engineering. Try to combine 2 or more features into 1 without losing sense of that variable. For example if you have a dataset of price prediction of used cars, and you have month_of_registration and year_of_registration as 2 of the features. You could combine them to form age variable and then drop the month and year variable. This is the most effective and non invasive way to reduce dimensionality.

2.) Second thing to do is feature selection. This can include any of the 4 techniques namely filter based, wrapper based, embedded and hybrid. This step also includes PCA if you want to use it.

3.) Even after doing above steps you get a large no of features, you can drop some less frequent categories from a feature. For example, if in a feature you have 2 categories and 1 one them is occurring only 1-2% of the time. You can drop that category to reduce the dimensionality.

Improve this answer
$\endgroup$
0
$\begingroup$

Tree Based models like RF and XGBoost can handle the feature space created by either of the methods you suggested above. Additionally, you can try to do feature selection (use those statistical tests) to eliminate 0 variance features and then feed the filtered dataset to your models.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.