Incorrectly applying random forest model?

Question

I am fairly new to random forest models (and data science in general), and was wondering if I am operationalizing the model I created correctly.

Context: I am creating a random forest model to predict employee voluntary turnover.

Problem: Test set has an accuracy of 97% (AUC .992[this seems too high], Precision: 88%, Recall: 99%) vs train set accuracy of 96%, and yet as we keep getting new voluntary terminations, their probabilities based on the model tend to be less than .1. I.E., indicating that they are not going to leave.

I guess that makes sense considering that as of the time when the model was created, those employees were still within the organization and therefore the model accurately classified them as with the company, but accurately classifying whether an employee is still with the organization or not is of no help, I need to be able to identify those employees that have a higher probability of leaving, which is what my understanding of what the RF model would do (I have done this before with logistic regression models).

Possible Explanations for this that I can think of:

1) Data set doesn't seem too imbalanced: 8054 non terms vs 2158 terms

2) Overfitting? But the test set doesn't drop substantially in accuracy

3) high correlated predictor variables?

Code Attached.

Thanks!

Answer 1

I think you correctly identified the issue: if your model tries to classify whether an employee stays or leaves, by definition every employee "stays" as long as they are an employee in the company.

A possible direction would be to design the response variable as "does the employee leave within a year?" (or any specific period of time). This way you can use the records of past employees who have left and past employees who have stayed across time, i.e. you can have several instances corresponding to the same employee at different times. You could add features such as "had a raise or promotion in the past 2 year" for instance.

Answer 2

This looks like a classic case of class imbalance,

1. You try oversampling the data.
2. Since you are targeting an event that is less likely to happen, It is good to optimize on high precision and you can compromise on recall.
3. Looks like you got lucky there, try using k-fold cross-validation while splitting the dataset into test and train, this looks a little odd because your test accuracy is higher than training accuracy. This might be because of several reasons which can only be diagnosed by looking at the datasets.