I'm working on a short data science project to compare the accuracy of different classification methods. The groups decided to use and compare Random Forest, Naive Bayes and SVM.

The dataset we are using has four categorical features. Each of which has a large number of unique values.

  • There are 16537 unique combinations of 17370 unique values in FeaureA.
  • There are 13860 unique combinations of 13852 unique values in FeaureB.
  • There are 3295 unique combinations of 29 unique values in FeaureC.
  • There are 1518 unique combinations of 29 unique values in FeaureD.

From what I've read the RF and NB algorithms should work fine with label encoding but SVM requires one hot encoding. However that would increase the number of features by ~35K. The performance cost seems like it would be significant. Ideally we would use the same encoding for all three algorithms. Would it be better to take the performance hit and try something like PCA for feature reduction?

  • $\begingroup$ Depends on the model. It is fine for RF but will not work for SVM, LR etc. $\endgroup$ – 10xAI Apr 16 at 17:18
  • $\begingroup$ maybe frequency encoding, be better, towardsdatascience.com/… $\endgroup$ – Nikos M. Apr 17 at 8:29
  • 1
    $\begingroup$ How many instances do you have? You would need a very large dataset to avoid overfitting with this level of diversity in the features. Also what do you mean by "X unique combinations of Y unique values"? It looks like each feature is itself a combination of several things, isn't it? Be aware that any feature value which occurs only once in the data is unusable. $\endgroup$ – Erwan Apr 17 at 19:05
  • $\begingroup$ The features are things like genre so you have one entry thats action, one thats action;indie, etc Feature A is publisher so nearly all the values are unique I suppose it would be best to eliminate that feature entirely then as it has little to no predictive value? $\endgroup$ – Jim Jones Apr 18 at 13:41

Tree Based models like RF and XGBoost can handle the feature space created by either of the methods you suggested above. Additionally, you can try to do feature selection (use those statistical tests) to eliminate 0 variance features and then feed the filtered dataset to your models.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.