2
$\begingroup$

My dataset has 140k rows with 5 attributes and 1 Attrition as target variable (value can either be 0 (Customer churn) or 1 (Customer Does not churn)). I divided my dataset in 80% training and 20% testing. My dataset is heavily imbalanced. 84% of my dataset has 0 as target variable and only 16% has 1 as target variable.

The feature importance of my training dataset is as follows:

ColumnA = 28%, ColumnB = 27%, AnnualFee- 17%, ColumnD - 17% and ColumnE - 11%

I initially wanted to do a very simple check of my model. After creating a Random Forest Classifier I tested the model on a dataset with just 5 rows. I kept all variables constant except Column AnnualFee. Below is a snapshot of my test dataset:

 Column A   Column B    AnnualFee   ColumnD ColumnE
 4500       3.9          5%         2.1      7
 4500       3.9          10%        2.1      7
 4500       3.9          15%        2.1      7
 4500       3.9          20%        2.1      7
 4500       3.9          25%        2.1      7

I expected that as annual fee increases the probability of customer churn also increases. But my rf.predict_proba(X_test) seems to be all over the place. I am not sure why this is happening:

I tried two different codes but the anomaly seems to be happening on both the codes:

Code 1:

rf = RandomForestClassifier(n_estimators = 400,random_state = 0, 
min_samples_split=2,min_samples_leaf=5,
                      class_weight = {0:.0001,1:.9999})
rf.fit(X_train, Y_train )

Code 2: Not My Code - Got it Online

from sklearn.model_selection import StratifiedKFold
from sklearn.model_selection import GridSearchCV
clf_4 = RandomForestClassifier(class_weight = {0:1,1:5})
estimators_range = np.array([2,3,4,5,6,7,8,9,10,15,20,25])
depth_range = np.array([11,21,35,51,75,101,151,201,251,301,401,451,501])
kfold = 5
skf = StratifiedKFold(n_splits = kfold,random_state = 42)

model_grid = [{'max_depth': depth_range, 'n_estimators': estimators_range}]
grid = GridSearchCV(clf_4, model_grid, cv = StratifiedKFold(n_splits = 5, 
random_state = 42),n_jobs = 8, scoring = 'roc_auc')
grid.fit(X_train,Y_train)

I would really appreciate any help on this!

$\endgroup$
  • $\begingroup$ What do you mean by the probabilities seem to be all over the place. A small change in the data should result in different probabilities. What is the accuracy obtained on your small dummy set? $\endgroup$ – JahKnows Apr 24 '18 at 2:17
  • $\begingroup$ It will be beneficial for you to atleast once Visualize your RF using graphviz.. $\endgroup$ – Aditya Apr 24 '18 at 3:57
  • $\begingroup$ @JahKnows Below is a snapshot of the probability distribution AT 5% probability of Churn = 47%, 10% = 48%, 15% = 49%, 20% = 50% and 25% probability of churn drop to 47%. I am not sure why the dip is happening at 25%. I would the probability of churn will increase from 20% to 25% $\endgroup$ – TigSh Apr 25 '18 at 18:06
2
$\begingroup$
  1. What do you mean by 'all over the place'? Have you tried actually using scoring measures such as accuracy scores, AUC, etc? These will play a vital role in determining how your model is performing. I'd suggest ROC AUC as to start off since accuracy will be pretty misleading due to the class imbalance.

  2. The class weights you used make little sense. You usually would use something like n_samples / (n_classes * np.bincount(y)) for your weighting, but since sklearn provides automatic balancing of classes, you should probably use that by setting class_weights='balanced': http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html

$\endgroup$
  • 2
    $\begingroup$ For future reference, please use the comments for asking questions to the OP. The answer is meant as a general purpose answer for future users stumbling across the same problems. $\endgroup$ – JahKnows Apr 24 '18 at 2:20
  • $\begingroup$ Below is a snapshot of the probability distribution AT 5% probability of Churn = 47%, 10% = 48%, 15% = 49%, 20% = 50% and 25% probability of churn drop to 47%. I am not sure why the dip is happening at 25%. I would the probability of churn will increase from 20% to 25% 2. I tried randomoversampling, oversampling, SMOTE but my recall and precision for minority class is always very low. I also tried class_weight = ‘balanced’ but the accuracy of my model was very low. I then randomly tried class weights = {0:.0001,1:.9999} and got accuracy = 59% $\endgroup$ – TigSh Apr 25 '18 at 18:23
2
$\begingroup$

You can directly enforce monotonicity. sklearns RF doesn't appear to support this (https://stats.stackexchange.com/questions/383423/how-to-enforce-a-monotonic-answer-in-a-single-feature-in-a-binary-classification), but XGBoost (https://xgboost.readthedocs.io/en/latest/tutorials/monotonic.html) and some others do.

Of course, improving the model in general is good (see Carlo's answer for some initial steps); and it's possible that your intuition for monotonicity just doesn't hold in reality.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.