# Random Forest Classifier Probabilities

My dataset has 140k rows with 5 attributes and 1 Attrition as target variable (value can either be 0 (Customer churn) or 1 (Customer Does not churn)). I divided my dataset in 80% training and 20% testing. My dataset is heavily imbalanced. 84% of my dataset has 0 as target variable and only 16% has 1 as target variable.

The feature importance of my training dataset is as follows:

ColumnA = 28%, ColumnB = 27%, AnnualFee- 17%, ColumnD - 17% and ColumnE - 11%

I initially wanted to do a very simple check of my model. After creating a Random Forest Classifier I tested the model on a dataset with just 5 rows. I kept all variables constant except Column AnnualFee. Below is a snapshot of my test dataset:

 Column A   Column B    AnnualFee   ColumnD ColumnE
4500       3.9          5%         2.1      7
4500       3.9          10%        2.1      7
4500       3.9          15%        2.1      7
4500       3.9          20%        2.1      7
4500       3.9          25%        2.1      7


I expected that as annual fee increases the probability of customer churn also increases. But my rf.predict_proba(X_test) seems to be all over the place. I am not sure why this is happening:

I tried two different codes but the anomaly seems to be happening on both the codes:

Code 1:

rf = RandomForestClassifier(n_estimators = 400,random_state = 0,
min_samples_split=2,min_samples_leaf=5,
class_weight = {0:.0001,1:.9999})
rf.fit(X_train, Y_train )


Code 2: Not My Code - Got it Online

from sklearn.model_selection import StratifiedKFold
from sklearn.model_selection import GridSearchCV
clf_4 = RandomForestClassifier(class_weight = {0:1,1:5})
estimators_range = np.array([2,3,4,5,6,7,8,9,10,15,20,25])
depth_range = np.array([11,21,35,51,75,101,151,201,251,301,401,451,501])
kfold = 5
skf = StratifiedKFold(n_splits = kfold,random_state = 42)

model_grid = [{'max_depth': depth_range, 'n_estimators': estimators_range}]
grid = GridSearchCV(clf_4, model_grid, cv = StratifiedKFold(n_splits = 5,
random_state = 42),n_jobs = 8, scoring = 'roc_auc')
grid.fit(X_train,Y_train)


I would really appreciate any help on this!

• What do you mean by the probabilities seem to be all over the place. A small change in the data should result in different probabilities. What is the accuracy obtained on your small dummy set? – JahKnows Apr 24 '18 at 2:17
• It will be beneficial for you to atleast once Visualize your RF using graphviz.. – Aditya Apr 24 '18 at 3:57
• @JahKnows Below is a snapshot of the probability distribution AT 5% probability of Churn = 47%, 10% = 48%, 15% = 49%, 20% = 50% and 25% probability of churn drop to 47%. I am not sure why the dip is happening at 25%. I would the probability of churn will increase from 20% to 25% – TigSh Apr 25 '18 at 18:06

2. The class weights you used make little sense. You usually would use something like n_samples / (n_classes * np.bincount(y)) for your weighting, but since sklearn provides automatic balancing of classes, you should probably use that by setting class_weights='balanced': http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html
You can directly enforce monotonicity. sklearns RF doesn't appear to support this (https://stats.stackexchange.com/questions/383423/how-to-enforce-a-monotonic-answer-in-a-single-feature-in-a-binary-classification), but XGBoost (https://xgboost.readthedocs.io/en/latest/tutorials/monotonic.html) and some others do.