0
$\begingroup$

Is it a good practice to evaluate a model on the training set (i.e. train a model on training set and evaluate the regression error/accuracy on the same training set) and compare the evaluation result with the model regression error/accuracy of cross validation (we do the cross validation on the same training set) and test set in order to check for overfitting/underfitting?

Since to my knowledge, we should never evaluate a model on the training set. However, I saw some lectures that seem to promote evaluating the training error.

$\endgroup$
1
$\begingroup$

Ok, let's be clear:

  • When we say that evaluation should never be done on the training set, it means that the real performance of the model can only be estimated on a separate test set.
  • It's totally fine to calculate the performance of a system on the training data, and it's often useful (e.g. to avoid overfitting). Of course the obtained result does not represent in any way the real performance of the system, so it's important to make sure that there's no confusion by mentioning it clearly.
$\endgroup$
0
$\begingroup$

No, you are right we should never evaluate our model on training dataset as you should check whether your model works not only gives desired results on the trained dataset but also the unseen dataset.

Training error is different it is the error that you get when you run the trained model back on the training data.We use it to see whether our model is giving generalized results.

Refer this to know about evaluating model.

$\endgroup$
2
  • $\begingroup$ Thanks for your answer. What do you mean with "run the trained model back on the training data"? do you mean make the trained model predicts the label of the target values of training data, and check for error/accuracy? $\endgroup$
    – glorian
    Aug 20 '20 at 8:54
  • $\begingroup$ To put it simply, training error is the error between the true labels and the predicted labels during training. $\endgroup$ Aug 20 '20 at 9:10
0
$\begingroup$

Although testing on all train-validation-test set are preferred for getting good view of over-fitting and under-fitting. you can think of cross validation as improvement of model on training dataset but it won't tell you how you're model behaved from start and testing is necessary to get idea how model will behave in natural settings.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.