1
$\begingroup$

I am trying to use random forest to select important variables out of 15K features and fit them into logistic regression. My evaluation is based on F1 score. Dataset 2 classes ratio are around: 99.5:0.5.

Here is where 15K features come from: Initally I have 2000 features, after dumminize them (only taking top 100 categories from categoritical variable) it became 16K. After removing zero variance it became 15K. I didn't want to remove near zero variance because of my class imbalance ratio is also very small. I tried removing near zero variance features before and it significantly reduces the number of features, however the logistic regression result was also not good.

However, after using grid search, random forest still has very low f1 score for cross validation. (less than 0.01)

Also I tried undersampling for the training data to make it 1:1, random forest still has bad F1 score for cross validation. :(

So I was thinking about not selecting the important variables and just fit all features to logistic regression.

Due to memory issue, I am not able to fit all 15K features into logistic regression directly but if I select important variables from random forest, they do not generalize for unseen test data.

Any ideas on how to solve the problem? I know one alternative could be using hashing so I could fit all 15K features.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

15K features is a lot of features. I'd bet your 0.01 F1 score is a direct result of having so many features. Without knowing more details about your data it is hard to suggest a more sensible alternative. I'd want it to be closer to 100 features rather than 15,000 though. Are these features extracted from text? N-grams?

Regarding your feature selection problem, the first step is to eliminate any features with no variance (all samples the same), then one can utilis various statistical methods for selecting features to retain those with high variance (sklearn has great libraries for this: Feature selection).

One could also utilize Ranked Guide Iterative Feature Elimination (RGIFE) for this purpose, which uses random forests: RGIFE - Ranked Guided Iterative Feature Elimination

P.S. Your class ratio is very skewed as well. Once you're all settled with a more reasonable dataset in feature space I'd consider either undersampling the larger class, oversampling the smaller class or a combination of both, and keep an eye on the resulting confusion matrices when performing your evaluation.

Improve this answer
$\endgroup$
4
  • $\begingroup$ To add onto DanJ's answer, if you find feature selection not working out, you can try dimension reduction using decomposition. You can read up about it on sklearn here. $\endgroup$
    – Impuls3H
    Apr 19, 2017 at 3:39
  • $\begingroup$ @DanJ Thank you Dan! So initally I have 2000 features, after dumminize them ( only taking top 100 categories from categoritical variable) it became 16K. After removing zero variance it became 15K. I didn't want to remove near zero variance because of my class imbalance. Also I tried undersampling to make it 1:1, random forest still has bad F1 score for cross validation. :) $\endgroup$
    – Alice
    Apr 20, 2017 at 3:10
  • $\begingroup$ @Impuls3H Yea I wanted to try PCA but haven't had a chance to try yet. Now it's time! Thanks!! :) $\endgroup$
    – Alice
    Apr 20, 2017 at 3:15
  • $\begingroup$ Yes most definitely try PCA first! There's plenty of other techniques if PCA doesn't wrk well on your data too, such as LDA and ICA. Try visualising the data and of course, setting up a pipeline to try the different methods. $\endgroup$
    – Impuls3H
    Apr 20, 2017 at 3:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.