2
$\begingroup$

For an employee population, I am trying to determine who among the employees are likely to get injured in the future based on 2 years worth of data.

Unlike in most machine learning problems where you try to predict based on unobserved data, I will be dealing with the same population since you do not see a lot of employee turnover in a short span of time.

I am using XGboost implementation. Of the whole population, my label tells whether the employee got injured or not. I used 70% of my data as my training set while tested the accuracy to the remaining 30%. I got a fairly decent accuracy rating. I was able to classify injured employees accurately by 90% (specificity). Although my negative prediction was only at 70% accuracy. my assumption is that those that were misclassified as injured although they were not in the last 2 years are those that are likely to get injured in the future or had been injured 2 years prior.

I now tried testing the algorithm to my whole population. I reshuffled my whole population data using the R code below hoping the algorithm won’t recognize observations that were both in the test and training data :

df2 <- df1[sample(nrow(df1)),]

In short, my training data is a subset of my full population (new/ final test data). When I compared the results, the training and testing error were almost identical.

Is it because, a large portion of my final test data already learned from the observations also found in my training data?

If my methodology is unacceptable, What other options can I implement so my test data won’t recognize that some of the observations included in it were already in my training data set?

$\endgroup$

1 Answer 1

1
$\begingroup$

The Out of Bag(OOB) error in XG Boost is considered as a Validation error(test error). During Hyper-parameter selection in XG boost, OOB error is measured for different hyperparameters. In XG Boost algorithm, the decision trees are constructed by a random sample of observations and the successive decision trees are built by giving more importance to the observation which the previous trees are misclassified. As this XG boost keeps increasing the number of decisions trees, the validation error decreases and saturates at some point.

In your case, you can consider this OOB error as test error instead of creating a separate test set.

The following plot explains this more accurately enter image description here

$\endgroup$
1
  • $\begingroup$ If I were to use OOB error, how would I code this using R to test for the whole population, or at a minimum, to the training set already evaluated? I'm not sure if I'm making sense. My understanding from the response is that I can evaluate using OOB error but I'm not familiar on how to execute this. I'm done hypertuning my parameters using MLR package. $\endgroup$
    – user123249
    May 25, 2018 at 18:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.