Is there any Python library that provides ready-to-use metrics to analyze the performance of a classifier for a multioutput-multiclass classification task? scikit-learn doesn't have this option yet (as stated in the documentation and in the corresponding feature request on GitHub).
$\begingroup$
$\endgroup$
2
asked Aug 12, 2015 at 23:14
-
$\begingroup$ I am still interested in this question. $\endgroup$– Franck DernoncourtJan 7, 2016 at 17:49
-
$\begingroup$ So any answer is welcome :) $\endgroup$– Franck DernoncourtJan 7, 2016 at 17:49
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Late answer! Sorry if I am repeating things you know...
I am sure somebody has worked on this formally, but least in my short industry experience, it might just be easier to roll your own solution here. I recently had to do this for classifying businesses (into restaurants, or dentists, or doctors, or amusement parks etc). Because calling a dentist a doctor was a relatively minor error compared to calling a doctor an amusement park, certainly all mistakes are not equal. My colleague decided to make a different ROC curve for each true class, and call things a "hit" if the classification was right, and a "miss" if the classification was wrong, and so he could leverage typical scikit-learn stuff, but it still only told part of the picture.
With all that in mind, we can still be quantitative. Perhaps you could decide on a misclassification matrix pentaly, where cell (i,j) is the penalty for calling an "i" a "j", with 0 penalties on the diagonal for correct classifications. Then you could at least compare models (at one particular set of thresholds) by their cumulative error or squared error etc.
It's already a little tough striking a tradeoff between false positives and false negatives in a binary classification problem - and it depends very heavily on your subject matter and what risks you are willing to tolerate. Do you want to be able to say "we aren't confident and therefore we won't guess"? This complicated things further for me, because we were trading off accuracy with automation, so "not sures" were also unfortunate, but in a different sense.
