I am trying to use random forest to select important variables out of 15K features and fit them into logistic regression. My evaluation is based on F1 score. Dataset 2 classes ratio are around: 99.5:0.5.
Here is where 15K features come from: Initally I have 2000 features, after dumminize them (only taking top 100 categories from categoritical variable) it became 16K. After removing zero variance it became 15K. I didn't want to remove near zero variance because of my class imbalance ratio is also very small. I tried removing near zero variance features before and it significantly reduces the number of features, however the logistic regression result was also not good.
However, after using grid search, random forest still has very low f1 score for cross validation. (less than 0.01)
Also I tried undersampling for the training data to make it 1:1, random forest still has bad F1 score for cross validation. :(
So I was thinking about not selecting the important variables and just fit all features to logistic regression.
Due to memory issue, I am not able to fit all 15K features into logistic regression directly but if I select important variables from random forest, they do not generalize for unseen test data.
Any ideas on how to solve the problem? I know one alternative could be using hashing so I could fit all 15K features.