Can you reuse observations from your train data in your final test data?

Question

For an employee population, I am trying to determine who among the employees are likely to get injured in the future based on 2 years worth of data.

Unlike in most machine learning problems where you try to predict based on unobserved data, I will be dealing with the same population since you do not see a lot of employee turnover in a short span of time.

I am using XGboost implementation. Of the whole population, my label tells whether the employee got injured or not. I used 70% of my data as my training set while tested the accuracy to the remaining 30%. I got a fairly decent accuracy rating. I was able to classify injured employees accurately by 90% (specificity). Although my negative prediction was only at 70% accuracy. my assumption is that those that were misclassified as injured although they were not in the last 2 years are those that are likely to get injured in the future or had been injured 2 years prior.

I now tried testing the algorithm to my whole population. I reshuffled my whole population data using the R code below hoping the algorithm won’t recognize observations that were both in the test and training data :

df2 <- df1[sample(nrow(df1)),]

In short, my training data is a subset of my full population (new/ final test data). When I compared the results, the training and testing error were almost identical.

Is it because, a large portion of my final test data already learned from the observations also found in my training data?

If my methodology is unacceptable, What other options can I implement so my test data won’t recognize that some of the observations included in it were already in my training data set?

Answer 1

The Out of Bag(OOB) error in XG Boost is considered as a Validation error(test error). During Hyper-parameter selection in XG boost, OOB error is measured for different hyperparameters. In XG Boost algorithm, the decision trees are constructed by a random sample of observations and the successive decision trees are built by giving more importance to the observation which the previous trees are misclassified. As this XG boost keeps increasing the number of decisions trees, the validation error decreases and saturates at some point.

In your case, you can consider this OOB error as test error instead of creating a separate test set.

The following plot explains this more accurately