2
$\begingroup$

I have a list of people, attributes about those people (height, weight, blood pressure, etc.), and a binary target variable called has_heart_issues. This data represents the full population of data, and I am trying to determine whether anyone who is listed as "No" for has_heart_issues is similar to the people who are listed as "Yes".

To answer this question, I split the data into training (70%) and testing (30%). I trained a random forest model on the training, and I tested it on the testing. The results are good, but I don't know how to apply to the population since I used most of it for training. Is there any way to apply the model to the full dataset (including the training) since I had labels for the full dataset to start with? Essentially, I am trying to determine whether any of the people were mislabeled.

Is it okay to apply the model to the training data to find the "mislabeled" records?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

There is exactly one thing you can check by examining the predictions on your training data. That is the numerical convergence of your model training routine. Any validation of model accuracy can only use holdout data or test data - that is the entire point of cross validation. Once the model architecture and hyperparameters have been optimized through n-fold cross-validation, the standard procedure is to train a single production model on the entire dataset. At that point, you've gotten all the information from the training set that you can.

Improve this answer
$\endgroup$
5
  • $\begingroup$ then how would I predict on the whole dataset if I train on 70% and test on 30%? or is it just not possible? in other words, is any data I train on essentially useless once it's been trained on? $\endgroup$
    – OverflowingTheGlass
    Mar 12, 2020 at 16:24
  • $\begingroup$ yes, any data you train on becomes useless for assessing anything about your model, except for assessing numerical convergence of the training optimizer. $\endgroup$
    – Dave Kielpinski
    Mar 12, 2020 at 17:56
  • $\begingroup$ I'm not trying to assess my model with it. I'm essentially trying to productionize my model and apply it to all my data. Is the answer still the same? Training data cannot have a model applied to it once the model is productionized? $\endgroup$
    – OverflowingTheGlass
    Mar 12, 2020 at 20:47
  • 1
    $\begingroup$ The entire purpose of cross-validation is to assess your model. In no case should you attempt to use any data used to train a model for assessing that model. Not sure how to make this clearer, and not too interested in doing so since you've neither upvoted nor accepted my answer. $\endgroup$
    – Dave Kielpinski
    Mar 13, 2020 at 16:36
  • 1
    $\begingroup$ I'm not trying to assess my model, as I said in my previous comment. I am trying to get actual predictions, actual results for people to act on. I can't upvote/accept your answer if it doesn't address my actual question. I may be too dense to understand, but that doesn't mean I can upvote/accept until I do. $\endgroup$
    – OverflowingTheGlass
    Mar 13, 2020 at 18:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.