2
$\begingroup$

I am fairly new to random forest models (and data science in general), and was wondering if I am operationalizing the model I created correctly.

Context: I am creating a random forest model to predict employee voluntary turnover.

Problem: Test set has an accuracy of 97% (AUC .992[this seems too high], Precision: 88%, Recall: 99%) vs train set accuracy of 96%, and yet as we keep getting new voluntary terminations, their probabilities based on the model tend to be less than .1. I.E., indicating that they are not going to leave.

I guess that makes sense considering that as of the time when the model was created, those employees were still within the organization and therefore the model accurately classified them as with the company, but accurately classifying whether an employee is still with the organization or not is of no help, I need to be able to identify those employees that have a higher probability of leaving, which is what my understanding of what the RF model would do (I have done this before with logistic regression models).

Possible Explanations for this that I can think of:

1) Data set doesn't seem too imbalanced: 8054 non terms vs 2158 terms

2) Overfitting? But the test set doesn't drop substantially in accuracy

3) high correlated predictor variables?

Code Attached.

Thanks! enter image description here

$\endgroup$
  • 2
    $\begingroup$ Please set code on the forum, not at the screenshot. $\endgroup$ – fuwiak Nov 7 '19 at 23:19
  • 2
    $\begingroup$ Are you taking into account that your data is right-censored, i.e. everyone will leave eventually you just haven't observed it yet? For example, someone might have joined the company a week ago. How can you be sure to label this employee "non term" compared to someone who left after 10 years with the company? You may want to look into survival analysis. $\endgroup$ – oW_ Nov 8 '19 at 17:03
  • $\begingroup$ Survival analysis is certainly an option, one that I have done before for this type of analysis, along with logistic regression. I decided to try an RF model this time simply due to its ease in implementation. However, given that tenure is also a variable in the model, this should be accounting for at least some variance in the outcome (i.e., termination), or at least I would think it would $\endgroup$ – riz Nov 11 '19 at 16:12
1
$\begingroup$

I think you correctly identified the issue: if your model tries to classify whether an employee stays or leaves, by definition every employee "stays" as long as they are an employee in the company.

A possible direction would be to design the response variable as "does the employee leave within a year?" (or any specific period of time). This way you can use the records of past employees who have left and past employees who have stayed across time, i.e. you can have several instances corresponding to the same employee at different times. You could add features such as "had a raise or promotion in the past 2 year" for instance.

$\endgroup$
-1
$\begingroup$

This looks like a classic case of class imbalance,

  1. You try oversampling the data.
  2. Since you are targeting an event that is less likely to happen, It is good to optimize on high precision and you can compromise on recall.
  3. Looks like you got lucky there, try using k-fold cross-validation while splitting the dataset into test and train, this looks a little odd because your test accuracy is higher than training accuracy. This might be because of several reasons which can only be diagnosed by looking at the datasets.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.