2
$\begingroup$

I have a random forest classifier that predicts 0 class about twice as often as class 1. It also predicts class 0 with higher probabilities than class 1.

It is not a imbalanced dataset. I tried setting class 1 weight to 100 and it seems to solve the problem, though I suppose it's not a correct solution :D K-NN gives the same problem. Since I changed y from 0 and 1 to B and A it started to predict second class more frequently. So can the problem be somehow connected to data type?

Code:

dataset = pd.read_csv('regtraining.csv')
X = dataset.iloc[:, :-5].values
y = dataset.iloc[:, 50].values

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2)
classifier = RandomForestClassifier()
classifier.fit(X_train, y_train)

y_pred = classifier.predict(X_test)

print(confusion_matrix(y_test, y_pred))

When I try multiple random states of train test split, one of predicted classes is always predicted much more frequently.

Edit: After some research I think the random forest splits so that the classes are predicted in an alphabetic order (A, B values for y give more of A but B, A values for y give more of B).

Thanks for your answers, I am new to machine learning :D

$\endgroup$
3
  • $\begingroup$ I am not concerned with a class being predicted more than another. I am concerned with what makes the "best" predictions. In this case there is a cutoff value that is being used to make the prediction. Is that cutoff value optimal for your problem? It needs to balance the costs of false positive/negative with the benefit of true positive/negative. Changing the cutoff value will change the predicted class. In your case, what are the best metrics for your problem and are the current prediction optimizing that metric? $\endgroup$
    – Craig
    Jan 20 '21 at 10:02
  • $\begingroup$ Might be because of data. Could you share the confusion matrix. Also, the split diagram with a Tree of depth 1. This might have some clue $\endgroup$
    – 10xAI
    Jan 21 '21 at 13:56
  • $\begingroup$ I added new info, thanks for your comments. $\endgroup$ Jan 23 '21 at 12:12
0
$\begingroup$

While prediction, If you have more date relevant to class 0 obviously this is the expected one.

But if your observation is like, the model predicts more data with class 0 as incorrect, then you model is kind of over-fitted.

To avoid these,

  • 57% and 43% is not kind of class imbalanced, but still you use class_weight with model.

  • Feature Engineering helps a lot with these kind of situations.

$\endgroup$
1
  • $\begingroup$ Hi, thanks for your answer. I don't think overfitting is the problem, as I lowered max depth to 1 and it's still the same. $\endgroup$ Jan 20 '21 at 9:52
0
$\begingroup$

It might be overfitting case. You can perform hyperparameter tuning with GridSearchCV or RandomizedSearchCV. Then you can check the performance of the model. If you still have bias in the prediction, you much do feature engineering and feature selection of the data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.