Differences between Test & Score results and calculated by Confusion Matrix (accuracy, sensitivity and specificity)

Question

We are using Orange and have 2 files (training and testing). We apply different Learners (kNN, AdaBoost...) and get the Evaluation Results. But we have some doubts about some options in the Test & Score window.

When we apply the option "Test on train data" does it generate for each Learner a model and then applies it to the training population?

And the same about the "Test on test data", does it generate the model with the training data and this option applies it to the testing population?

Because when we get the confusion matrices and we calculate accuracy, sensitivity and specificity the values differ from the ones in the Evaluation Results.

Answer 1

Test on train data uses the whole data set for training and then for testing. This method practically always gives wrong results.

So, yes, in this case, the model is learned and tested on the same data set. In most cases, the model learned this way will be overfitted and will perform poorly on unseen data (e.g., your testing dataset).

Test on test data: the above methods use the data from Data signal only. To input another data set with testing examples (for instance from another file or some data selected in another widget), we select Separate Test Data signal in the communication channel and select Test on test data.

So, the answer to the second question is also "yes:" The model is trained on the data labeled Training in you diagram and then tested on the Testing data. The "quality" of the data and the way it was split in the training and testing set will influence the performance of the model.

Orange documentation on "Test & Score":

Answer 2

Test on train data gives you good results but when tested against the reality, majority of time it fails as test on train data does the testing on train data itself and results overfitting.

You should not rely on test on train data. To make your model robust to unseen data, I would recommended to use cross validation techniques like k-fold.It incorporates majority of the variance into your model and so in reality, model able to predict the outcome as intended. The accuracy will get reduced marginally as compare to test on train data but it makes the model more robust and stable.