I am looking into creating a model to predict whether an item is "Very Good", "Good", "Bad" or "Very Bad".

After I fit the training data to the models, comparing the accuracy of the models during test stump me: should it matter if a model misclassified a G to VG while the other G to VB? What about a model that has two misclassifications of one level away versus another model with only one misclassification but three levels away (eg VG to VB)?

Any guideline on what is the common approach? Also, my thinking at the moment is that this should be a regression problem, but I'm happy to be corrected if I should approach this labeling of datasets more as a classification problem.


Your classes express a certain order. You can classify apples to, say, "green", "red" or "yellow", and then every disagreement with a reference set is equal. After all, colours express no order. So as you already suggested, I would certainly use regression. Assume that the classes could be distributed as something like this:

  1. Very bad = 0 - 0.25
  2. Bad = 0.25 - 0.50
  3. Good = 0.50 - 0.75
  4. Very good = 0.75 - 1.00

Now, the mismatch of Very good vs. Bad is at least 0.25, where is must be at least 0.50 with Very good vs. Very bad, which gives a better and more honest impression of the performance of your model.

  • $\begingroup$ Is it valid to make such assumptions? From my training set, the distribution is definitely not equal as in your assumption (btw this variable is a response from survey, so a lot of subjectivity would come into the answers). So does it mean I should use the distribution of the training set to create a weighted distance measure? $\endgroup$ – Ricky Jun 6 '15 at 15:38
  • $\begingroup$ Of course. If it's really skewed, I would pick different margins, just make sure they are empirically grounded. Mine were example margins :) $\endgroup$ – lennyklb Jun 9 '15 at 9:14
  • $\begingroup$ what you choose as the margin may even depend on what errors matter more and which matter less as per your requirements. eg say we have an item X for which if label is B or BV, we would be discarding it else keeping it. So in this case the distinction between B and BV is less important then distinction between G and B. So your mismatch score between B and BV should be less than that between B and G $\endgroup$ – Shagun Sodhani Jun 12 '15 at 8:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.