I read that a confusion matrix is used with image classification but if I need to draw it with image captioning how to use it or can I draw it in the evaluation model phase for example if yes how can I start?


1 Answer 1


There's a confusion: a confusion matrix is a standard tool for evaluating a classification task, i.e. one where the target is a categorical variable. The confusion matrix is a table which allows observing the number of test instances which have true class X and are predicted class Y, for every class X and Y. This is practical only with a small number of classes of course, otherwise the confusion matrix is not readable.

The task of image captioning is not classification. The target is unstructured data (text), not a categorical variable with a finite set of possible values. Therefore it requires a different (and more complex) evaluation method. It's often similar to machine translation, based on a measure of similarity between the gold standard caption and the predicted caption. Usually one should use the state of the art evaluation method, i.e. the method used in recent papers published on this task.

  • $\begingroup$ Thanks a lot but if I need to take for example the results I got from captioning and predict if it is true or not based on the original caption > like "boy" how can I interpret the confusion here I mean it consists of 4 values (TP,TN,FP,FN) i mean the difference between them, please $\endgroup$
    – Lei
    May 18, 2022 at 10:16
  • $\begingroup$ @Lei Well in general it's not always possible to say whether a caption is correct or incorrect, it can often be partly correct. This is why standard captioning evaluation is not done this way. Do you mean that you are able for every caption in your data to categorize in this way? If so you could calculate an accuracy score, but it's a bit unusual. Obviously you cannot have the notion of gold positive or negative, because the gold standard caption isn't binary. This is why the confusion matrix doesn't make sense. $\endgroup$
    – Erwan
    May 18, 2022 at 10:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.