Evaluating Models that Return Percentage Present of Multiple Classes

Question

If there is a model that returns a vector of the amount of different classes present in the data as percentages, what would be a good way to evaluate it (with charts and/or statistics)?

Say, for example, that a batch of pond water contains 30% Bacteria1 and 70% Bacteria2 (data is [0.3, 0.7]. Our model returns 35% Bacteria1 and 65% Bacteria2 (output is [0.35, 0.65]). How would we evaluate the accuracy of this model?

Am I right in thinking that we can't use things like confusion matrices or ROC/AUC curves because this isn't a classification problem? I'm not sure if there exist other metrics like these ones for this kind of problem though.

Answer 1

If I understand correctly you're asking about the case where the model predicts a probability distribution or similar.

Assuming one has a test set with the true distribution for every instance, I think the most direct way to evaluate such a model is with a distance/similarity measure between distributions, for example KL divergence or Bhattacharyya. The distance is calculated between the predicted and true distribution for every instance, then the value is aggregated across instances (typically taking the mean).

You're correct that this is not classification, so classification evaluation techniques wouldn't apply (unless one is interested only in which class has the highest predicted probability in the distribution).