2
$\begingroup$

Edit: It turned out that I had an error in my function to compute the combined probabilities (a typo that changed the behavior of my function quite a bit without giving me an error message). Without that, my classification is much more satisfactory. Also, I don't use softmax anymore because my probabilities are normalized anyway. So the question is basically solved. If anyone has a good idea how to improve my hierarchical classification algorithm, I'm happy to hear it, though!

I have a data set with tf-idf/bm25 weighted words (I'm trying both approaches) with several classes that I try to predict. The classes have a hierarchical structure with 2 levels. A part of the hierarchy is shown in the graph. In total, there are 20 classes on the first level and 111 leaf nodes. Some parent nodes don't have any child nodes. 2 level hierarchy

I would like to fit a multi-class classifier for every parent node (so considering only the part of the hierarchy shown above, that would be 3 classifiers, one for the first level and two for the parent nodes that have children), which is called the “Local classifier per parent node approach” according to Silla & Freitas, 2011. How I understand it, the classifier for the first level is trained using the whole training set, while the classifiers for the second level are trained only using examples with labels that belong in these classes (so the classifier for the children of Catering is trained with only training examples that are labeled as Catering).

I planned on using Multinomial Naive Bayes and Support Vector Machines from scikit-learn as my classification algorithms. After training, I obtain log probabilities for the test set.

What I'm unsure about is how I should combine the probabilities that I obtain with my classifiers. I thought that I could just multiply the probability of the first level with the probabilities in the second level (or add the log probabilities) to get leaf node probabilities.

My reasoning behind that is that if a mistake is made on the first level, so that the true level 1 class has a relatively low probability, then it can be still compensated by a high probability in a child node.

After obtaining all probabilities, I would have normalized the leaf node probabilities so that they add up to 1 with softmax. However, when doing that, I get very similar probabilities for every leaf node, so that the highest leaf node probability is only a tiny bit higher than the probabilities of the other leaf nodes.

That makes sense in a way because I force every second level classifier to give me probabilities for only the children of that classifier and those add up to 1. So even if my test example does not belong to the parent node in question, one of the children classes is going to get a relatively high probability.

How can I avoid the problem of leaf node probabilities that don't discriminate enough? Or is there some other error in my reasoning that led to this result?

$\endgroup$
  • $\begingroup$ Can you give an example (say on the small subset of outcomes you've displayed) of the kind of scores/probabilities you get / expect to get? I'm not clear exactly what order you're thinking of doing things. $\endgroup$ – Ben Reiniger Mar 1 at 14:42
  • $\begingroup$ I hope I understand your question correctly, but I expect to get a probability for every leaf node, where most leaf nodes (preferably all but 1) have a probability of nearly 0 and a small number of leaf nodes have a probability near 1. In any case, the highest probability determines the class that is assigned to the document. At the moment, I get probabilities that are nearly equal for all leaf nodes, so in the small example above, most of my maximum leaf node probabilities are around 0.18 (that's only a tiny bit higher than if all leaf nodes had exactly the same probability). $\endgroup$ – Amelia Bones Mar 1 at 14:56
  • 1
    $\begingroup$ If I compute the F1 score for the resulting classification, I get something like 0.6, which is only a bit lower than if I use a flat classification approach. But the whole purpose behind a hierarchical classification algorithm is that you use the information that is provided by the hierarchy. I'm therefore looking for ways to improve how I use that hierarchy. $\endgroup$ – Amelia Bones Mar 1 at 14:56
  • 1
    $\begingroup$ Actually, these very similar probabilities are the result of a SVM (which I obtained by setting probability = True and by using the method predict_log_proba for the test set). I didn't try NB yet. I used softmax after multiplying my probabilities, so there are NOT on the individual model level. The probabilities on the individual model level are already scaled by default. $\endgroup$ – Amelia Bones Mar 1 at 15:21
  • 1
    $\begingroup$ No, I got actually rather nice probabilities for my first hierarchy level. The class with the highest predicted probability had in around 3/4 of the cases a probability higher than 0.6. So my first level is rather decisive. That's also why I was so surprised that my leaf nodes had probabilities that are this similar $\endgroup$ – Amelia Bones Mar 1 at 15:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.