My dataset has around 1000 features and 30k rows.

All the feautres have value either 1 or 0. My target variable is Size which 3 classes : Small, Medium and Large. I have around 5k "small" data points, 9k "Medium" data points and 20K "Large" data points. I want to find feature importance. I am not sure which algorithm will work best for features with only 1s and 0s. I tried Chi2 but accuracy is very low:

#find best scored 10 features
select_feature = SelectKBest(chi2, k=10).fit(X_train, Y_train)

X_train_2 = select_feature.transform(X_train)
X_test_2 = select_feature.transform(X_test)

clf_rf_2 = RandomForestClassifier(n_estimators=1000, n_jobs=-1, random_state = 100)      
clr_rf_2 = clf_rf_2.fit(X_train_2,Y_train)
ac_2 = accuracy_score(Y_test,clf_rf_2.predict(X_test_2))
print('Accuracy is: ',ac_2)
cm_2 = confusion_matrix(Y_test,clf_rf_2.predict(X_test_2))

My accuracy is only 58% and the model is only good in predicting "Large" data points. Below is the Confusion Matrix

      Small   Medium   Large
Small  6      17       3000
Medium 9      9        2000 
Large  31     32       7500

I mostly have 2 questions:

  1. I am not sure which algorithm will work for best features with only 1s and 0s
  2. How do I get higher accuracy using chi2 (predict Small and Medium class better)

Thanks! I would really appreciate any input on this.

  • $\begingroup$ Would the result be better if you use larger k? $\endgroup$ – user12075 Sep 21 '18 at 17:23
  • $\begingroup$ I have tried k = 100 but the accuracy didn't improve much $\endgroup$ – TigSh Sep 21 '18 at 17:24
  • $\begingroup$ how about lightgbm? $\endgroup$ – ipramusinto Sep 21 '18 at 18:59
  • $\begingroup$ Just to check for imbalance fuckery, try subsampling your medium and large classes such that you will have an equal number of samples in all classes (5k) and rerun your models. If performance does not improve then you should look at your data and how to "improve" it. $\endgroup$ – user2974951 Sep 25 '18 at 6:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.