6
$\begingroup$

A training set has five classes including:

"label-A", "label-B", "label-C", "label-D", "others"

But the problem is much simpler - it is to determine whether each input belongs to "label-ABCD" or "others". In this case, there are two solutions to solve this problem in my mind.

Solution 1: Train a 5-classes classifier, when the classifier predicts the input as "label-A" or "label-B" or "label-C" or "label-D", we relabel it as "label-ABCD".

Solution 2: Train a 2-classes classifier, we relabel the data as "label-ABCD" which is labeled as "label-A" or "label-B" or "label-C" or "label-D". And then it becomes a binary-classification problem.

My questions are:

  1. Which way can the model get a better performance in "theorem"?

  2. In real case, these two cases get almost the same performance by a CNN classification model, and I am wondering if I adopt a weaker classifier like C4.5, Naive Bayes, SVM...which method will win?

Thanks!!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
8
$\begingroup$

The greater the number of output nodes the higher complexity you will add to your model. This means that given a fixed amount of data, a greater number of output nodes will lead to poorer results. I would use a ABCD vs. others strategy.

Instead of conditioning your model to learn the distributions of the class A, B, C and D separately you will combine them. This means that is A and B are different in some way, but this difference is irrespective of the classification with "others" then there is no need to learn that distinction.

For example: if you want to detect dog, cat, human with features such as weight, height and number of legs. The number of legs feature will have relatively low importance, because cats and dogs will likely all have 4 legs. However if I want to classify cat/dog vs humans, then the number of legs will be the most important feature. It might be the only feature you need.

One caveat may be severe class imbalance. By combining your classes in this way you may end up with an over representation of ABCD. You can use techniques such as anomaly detection to train a model on your ABCD data and then detect whether a novel instance falls within this distribution, or is an outlier in which case you would label it as "others."

Improve this answer
$\endgroup$
9
  • $\begingroup$ In my opinion, if there is a zoo only contains tigers and lions, and given a picture of animals, we want to predict whether the animals is belongs to the zoo. If the input is liger(has lion mom & tiger dad), it has both lion and tiger's features so the model may confuse and predict it as one of the zoo's member, however, if the model has ability to identify lion and tiger individually, then the model may know two things, first liger is not a lion, second liger is not a tiger. Finally we know that liger is not belongs to the zoo, because it is neither a lion nor a tiger. $\endgroup$
    – PoCheng.Lin
    Commented Feb 9, 2018 at 10:10
  • $\begingroup$ So my conclusion is, multiclass model may understand more detailed of each class, so it can figure out the problem above. $\endgroup$
    – PoCheng.Lin
    Commented Feb 9, 2018 at 10:10
  • $\begingroup$ If we put all the class together (ABCD), does it extended a problem that the model can only extract the ABCD's feature roughly. Once we have the instance of A-. B+, C+ etc..., then The model may predict them as label-ABCD $\endgroup$
    – PoCheng.Lin
    Commented Feb 9, 2018 at 10:10
  • $\begingroup$ Considering your first comment. Should the liger fall within the zoo which is comprised entirely of tigers and lions? Moreover, consider the strength of machine learning, I don't think even a human would be able to perform this task reliably. But yes your conclusion may understand more details about each class, but in doing so it learns difference between classes that it might not otherwise need to consider if you would be to group the classes. Such as my dog/cat vs. others example. $\endgroup$
    – JahKnows
    Commented Feb 9, 2018 at 11:01
  • 1
    $\begingroup$ Are examples from a similar distribution of A-, B+, C+ represented in the training set labelled as "others". If not, this is just a wildcard, it's very hard to predict. Don't forget the training data should be collected from the same distribution as the data with which you will apply your model. $\endgroup$
    – JahKnows
    Commented Feb 9, 2018 at 11:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.