0
$\begingroup$

It is a completely theoretical question. What if the training data and the prediction data are the same? Will it impact the results if I use the same data for model predictions that I have used for training?

$\endgroup$
1
$\begingroup$

Yes, it will impact the results.

In the best case your model will not overfit at all so you could expect the same performance between the training set and a previously unseen set (= normally the testing set). In case your model overfits even a little bit, then your results on the training set will be higher (better) than on unseen data.

The impact depends on the classifier/hyperparameters you are testing. For instance, if you are using a KNearestNeighbors classifer with k=1, then the impact will be extreme as you will have 100% accuracy on whatever training set.

$\endgroup$
2
  • $\begingroup$ A difference in performance between training & test set is practically always expected, it has a specific name (generalization gap) and it does not signify overfitting in itself; the telltale signature of overfitting is that your training error continues to decrease while your validation error starts increasing. So, even if there is no overfitting, the performance metrics obtained from the training data alone are practically always expected to be better than the validation (and test) ones. $\endgroup$ – desertnaut Sep 28 '20 at 12:15
  • $\begingroup$ I agree and did not say that a difference in performance between training and testing set signify overfitting. But overfitting will lead to such a difference. Then, even if the model does not overfit at all, we could expect similar performances. I didn't find useful to get into the details here, but yes, there are other implicit assumptions, so the "could". $\endgroup$ – etiennedm Sep 28 '20 at 13:34
1
$\begingroup$

In addition to etiennedm's answer:

Informally, this is the equivalent of making a student take a test while giving them all the answers.

The goal of evaluating a model on an unseen test set is to measure the ability of the model to predict on different instances, in the same way that a school test is supposed to check if the student can solve new problems based on the knowledge they acquired. If the student is provided with all the answers, the result of the test will only show that they can read and copy the answer, and this is not very useful. Same thing for a program, it's pointless to prove that it can store and copy answers, and there's no need for Machine Learning to achieve that.

Evaluating on an unseen test set is meant to measure how well the model generalizes (i.e. actually "learns something") from what it has seen in the training data to answer new problems. Using the training instances as test set completely defeats this purpose, since the model doesn't need to generalize anything to answer these instances.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.