0
$\begingroup$

Currently, I have trained my model through 5-fold cross validation with very small amount of the sample (n=100).

I used whole data set to train and got quite low performance in terms of accuracy, which is bit higher than 70%.

However, if I put my data which was used for training back to trained model to validate, it gives me higher accuracy (80%).

So, my question is it okay to say that I have verified my trained model using training set and got 80 of accuracy? or should I have to stick with 70 % of accuracy that I received from 5-fold cross validation?

$\endgroup$

2 Answers 2

0
$\begingroup$

Scoring your model using the training data is not a best practice. The reason is that you have already used this data to develop the "guess" at an accurate model.

This is akin to saying - 'Pick a random number between 1 and 10 (20 times)', here's a list of the last 10 numbers I picked to get you started. You pick 20 new numbers and come up with 5 correct (25%). What you are suggesting is that instead of reporting the 25% accuracy, that you would throw the first 10 back into the mix and report a 50% accuracy because now you've gotten 15 out of 30 right.

So there are two different things here. The MSE of your initial model against the data you gave it - provides you and understanding of how well the model demonstrates history. While the MSE (or other error metric) of the 'test' data shows how well you can predict the future.

$\endgroup$
0
$\begingroup$

No, its similar to leaking the question paper for your final exams. And always trust your Cross val. Check your classes and try extracting features and check if it performs better

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.