Currently reading Learning from Little: Comparison of Classifiers Given Little Training

In 3 Experiment Results, the following graph is shared:

The experiment is described as follow

We begin by examining an example set of results for the average TP10 performance over all benchmark tasks, where the training set has P=5 positives and N=200 negatives.We vary the number of features selected along the logarithmic x-axis.

I understand this as "we use a training set of 205 elements, 5 being positives and the 200 remaining being negatives". But looking at the results, Naive Bayes using Information Gain with a few features, and Multinomial Naive Bayes using Bi-normal separation with several hundred features both end up with 6.5 true positives in the top 10 (The TP10 metric is the number of true positives found in the 10 test cases that are predicted most strongly by the classifier to be positive).

I would have assumed that the models being closest to 5/10 would be the most accurate, but reading their results, it looks like the higher the better. Thus, it feels like I overlooked and misunderstood something. Can somebody enlighten me on this issue ?

Thank you


Of course it is impossible to have a higher than $100\%$ true positive rate in your results.

The author does explain how he arrives at his results, but I must admit it is not at all obvious. In the subsection Performance Metrics of section 2 Experiment Protocol the author explains that the metrics are calculated as

The TP10 metric is the number of true positives found in the 10 test cases that are predicted most strongly by the classifier to be positive.

This mean that the TP10 is only calculated using the testing set. The author also explains in the subsection Experiment Procedure of section 2 Experiment Protocol states

The condition on the second loop ensures that there are 40 positives available for training plus at least 10 others in the test set (likewise, 200 negatives for training and at least 50 for testing)

So we see that there is actually a minimum of 10 testing instances with which the TP10 is computed.

  • $\begingroup$ thank you for the reply. If I understand, the model was trained using 205 elements, 5 of which were positive. Then the results shown on the graph are from the algo being run on the test dataset, which has at least 10 positives. I was confused by the procedure saying that 40 positives are available when the graph specifies 5, but alright. Now, the best score of TP10=6.5 means that out of the 10 most strongly predicted positive elements, 3.5 where actually false. it's probably not too wrong to assume that lesser predictions have an even worse rate. Looks pretty weak and unusable to me :S $\endgroup$ Sep 14 '18 at 2:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.