I am fairly new to random forest models (and data science in general), and was wondering if I am operationalizing the model I created correctly.
Context: I am creating a random forest model to predict employee voluntary turnover.
Problem: Test set has an accuracy of 97% (AUC .992[this seems too high], Precision: 88%, Recall: 99%) vs train set accuracy of 96%, and yet as we keep getting new voluntary terminations, their probabilities based on the model tend to be less than .1. I.E., indicating that they are not going to leave.
I guess that makes sense considering that as of the time when the model was created, those employees were still within the organization and therefore the model accurately classified them as with the company, but accurately classifying whether an employee is still with the organization or not is of no help, I need to be able to identify those employees that have a higher probability of leaving, which is what my understanding of what the RF model would do (I have done this before with logistic regression models).
Possible Explanations for this that I can think of:
1) Data set doesn't seem too imbalanced: 8054 non terms vs 2158 terms
2) Overfitting? But the test set doesn't drop substantially in accuracy
3) high correlated predictor variables?