2
$\begingroup$

This is a binary classification task, I have 15K 1's and 11K 0's (target)

I have tried the following:

X = feature_cols 
y = department_wise[['Threshold']]
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.2, random_state=1)
model = RandomForestClassifier()
model.fit(X, y)
predicted_labels = model.predict(X_test)

X_test predicts only 0 and the accuracy comes around 88%. I don't understand why since my data set is not even imbalanced. Whatever other classifier I am trying, it shows the same result with high accuracy. Please let me know where I am going wrong.

$\endgroup$
  • 1
    $\begingroup$ You may want to use F1 score to evaluate your results. $\endgroup$ – Media Jun 21 at 5:45
  • $\begingroup$ Thanks. Okay, i will but I am more concerned for the fact that it is just predicting one class i.e 0 when i print predicted_labels. $\endgroup$ – tired coder Jun 21 at 5:52
  • 2
    $\begingroup$ That is why I've mentioned try out F1. If you use it, you will be able to see whether it is always predicting zero for all or not. Your test data or your train data may be biased to a specific class. $\endgroup$ – Media Jun 21 at 5:58
  • $\begingroup$ Okay. Trying it out. $\endgroup$ – tired coder Jun 21 at 6:06
  • $\begingroup$ It's impossible to obtain 88% accuracy if the model predicts only 0s on a dataset which contains 57% of 1s. If it predicts only 0s, its accuracy will be exactly the proportion of zeros (around 43%). $\endgroup$ – Erwan Jun 21 at 13:03
1
$\begingroup$

It is not unusual that some method (given some data) predict only one class (meaning probabilities are all larger or smaller than 0.5). I had this case recently with a logistic regression application. The reason usually is that you don‘t have good features (x) to predict y. So first thing: have a look at correlation of X,y.

What you can do to improve your fit is basically to use boosting. Boosting is also tree based and similar to RF it is also an ensemble technique. But boosting gives higher weight to observations which are hard to predict. LightGBM or Catboost are good routines for boosting.

In my case, Logit predicted only one class with an AUC of about 0.3. LightGBM was much better and much more balanced in terms of prediction with an AUC of about 0.7.

You could also try Logit with L1 regulation (Lasso). Maybe some of your features are not very helpful in making predictions. Lasso shrinks these features, which helps to make okay predictions.

$\endgroup$
  • $\begingroup$ Thank you for this information! $\endgroup$ – tired coder Jun 25 at 12:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.