The accuracy of a random forest algorithm is nearly 1, how do I solve this problem? (with updates)

Question

I have a problem with a random forest algorithm, I'm firstly explaining the situation and then I'll ask questions.

I have a dataset of 10000 raws x 40 columns, 39 of them are features and 1 contains the labels. The 10000 raws represent different people and the label column shows that 7570 people satisfy a certain proprierty "0" and 2430 satisfy a proprerty "1". I have written a random forest algorithm and my goal is to train the model in order to recognize, considering the 39 features I've already mentioned, if a specific person satisfies property "0" or "1". The code is the following:

model = RandomForestClassifier(max_depth=1, n_estimators=250, random_state=0)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
print(f"Accuracy: {accuracy:.2f}")

It returns the value

Accuracy = 0.91

Then I computed the precision matrix:

from sklearn.metrics import confusion_matrix
cm = confusion_matrix(y_test, y_pred)
print(cm)

The result obtained has been the following:

Confusion matrix:
[[754   0]
 [90  156]]

I have also plotted the learning curves writing the following code block:

plt.figure()
plt.plot(train_sizes, train_scores_mean, 'o-', color='r', label='Training score')
plt.plot(train_sizes, test_scores_mean, 'o-', color='g', label='Cross-validation score')

plt.fill_between(train_sizes, train_scores_mean - train_scores_std, train_scores_mean + train_scores_std, alpha=0.1, color='r')
plt.fill_between(train_sizes, test_scores_mean - test_scores_std, test_scores_mean + test_scores_std, alpha=0.1, color='g')

plt.xlabel('Training examples')
plt.ylabel('Score')
plt.title('Learning Curves for Random Forest')
plt.legend(loc='best')
plt.grid(True)
plt.show()

In the end I have considered the features importance writing the following code block:

importances = model.feature_importances_
indices = np.argsort(importances)[::-1]
sorted_titles = [titles[i] for i in indices]

plt.figure()
plt.title("Features relevance")
plt.bar(range(X.shape[1]), importances[indices], align="center")
plt.xticks(range(X.shape[1]), sorted_titles, rotation=90)
plt.xlim([-1, X.shape[1]])
plt.xlabel("Feature")
plt.ylabel("Relevance")
plt.show()

After explaining everything I've done it's time for questions. The first thing I don't understand is why if I consider a value max_deph > 1 the accuracy becomes greater as the value of max_depth increases, the accuracy reaches 100% for values of max_depth greater than 10. Such a high value for the accuracy clearly is telling me something is not working correctly, but I'm unable to understand what it is. Can please someone help me? Is there any overfitting problem? Is there a problem with the dataset? Am I considering too many features? I'm looking for anyone who can help me, even if you cannot, thanks for having read my question! ;)

PS: If something is not clear, please tell me and I'll clarify it

--- UPDATES ---

As suggested in an answer, due to the fact the dataset is unbalanced, I set class_weight = 'balanced' in model definition:

model = RandomForestClassifier(max_depth=1, n_estimators=250, random_state=0, class_weight=balanced) 

and now the Confusion matrix is like:

Confusion matrix:
[[1464  44]
 [  40 452]]

Then, I printed the classification_report obtaining:

And finally I printed the ROC curve obtaining the following graph:

What do you think? Meanwhile I'll try other models like XGBoost or a DNN and see what happens.

Answer 1

To answer your questions: increasing the max depth of a Random Forest algorithm may increase complexity. Even though your 91% accuracy for the code you listed is test accuracy, this still could mean that your model is not doing well, in a sense:

Your dataset is imbalanced. Accuracy is not such a good metric to use on imbalanced datasets (some may consider it not that informative of what your model is doing in general). Instead, look at your model's predictive performance with other metrics. For example, you could try recall, precision, F1, AUC, AUPRC (area under precision recall curve), or Brier score, etc. These could give you a better idea of how your model is performing.

In some way, you could kind of say that your model is "overfitting": it could be not learning very well, and just predicting a bunch of 0's (though if it was just predicting all zeros, accuracy would be around 76% instead of the 90 something percent that you have now). As you are using Python, you could look as Classification Report from sklearn.

I would also recommend for you to do everything in a pipeline (sklearn or imblearn in Python). These would basically prevent data leaks. So, you could see what your model performance is like using a pipeline. If you are not doing it already, you could try scaling numerical features, one hot encoding categorical features, or other preprocessing / data processing techniques. If you look at other metrics besides accuracy and see that your model is not performing well, you could try other models: XGBoost, LightGBM, SVM (try different kernels), neural network (maybe, though the dataset may be kind of small for one).

Random Forest takes features into account when building itself; it can kind of do its own "feature selection". When trying other models, you could try to implement L1 regularization (which sometimes shrinks feature coefficients to zero, taking features into account).

To improve the predictive performance of your model (in terms of "more informative" metrics such as AUC or Brier score or F1), you could do hyperparameter tuning. RandomizedSearchCV with sklearn would be a good choice in Python. You could also see what a "baseline" model of yours would do with no hyperparameter tuning, but just k-fold cross validation, to get a sense of a baseline.

Answer 2

I guess the question is whether this is too good to be true, and if so what could be leading to it.

Domain knowledge is the first suggestion: if you know others have modeled this or similar problems, what kind of performance have they achieved? Is your score so much higher than that as to be suspicious?

Specific to your case, you get 91% accuracy with a random forest of stumps. Those are easy enough to understand, so print out (some of) the trees, and check their individual performance, and what feature they use. This is likely to expose one of the common gotchas: a feature that wouldn't be available when applying the model to new data (in the extreme case, leaving your target as an independent variable, though then you should have 100% accuracy instead of 91%). (You still need some domain knowledge to identify whether the splitting feature(s) are cheating in this way.) More generally (when you don't just have stumps), feature importances can surface such problematic features.

Another common gotcha is incorrect train/test splitting, or otherwise leaking test information into the training set/model.