For an employee population, I am trying to determine who among the employees are likely to get injured in the future based on 2 years worth of data.
Unlike in most machine learning problems where you try to predict based on unobserved data, I will be dealing with the same population since you do not see a lot of employee turnover in a short span of time.
I am using XGboost implementation. Of the whole population, my label tells whether the employee got injured or not. I used 70% of my data as my training set while tested the accuracy to the remaining 30%. I got a fairly decent accuracy rating. I was able to classify injured employees accurately by 90% (specificity). Although my negative prediction was only at 70% accuracy. my assumption is that those that were misclassified as injured although they were not in the last 2 years are those that are likely to get injured in the future or had been injured 2 years prior.
I now tried testing the algorithm to my whole population. I reshuffled my whole population data using the R code below hoping the algorithm won’t recognize observations that were both in the test and training data :
df2 <- df1[sample(nrow(df1)),]
In short, my training data is a subset of my full population (new/ final test data). When I compared the results, the training and testing error were almost identical.
Is it because, a large portion of my final test data already learned from the observations also found in my training data?
If my methodology is unacceptable, What other options can I implement so my test data won’t recognize that some of the observations included in it were already in my training data set?