Edit: It turned out that I had an error in my function to compute the combined probabilities (a typo that changed the behavior of my function quite a bit without giving me an error message). Without that, my classification is much more satisfactory. Also, I don't use softmax anymore because my probabilities are normalized anyway. So the question is basically solved. If anyone has a good idea how to improve my hierarchical classification algorithm, I'm happy to hear it, though!
I have a data set with tf-idf/bm25 weighted words (I'm trying both approaches) with several classes that I try to predict. The classes have a hierarchical structure with 2 levels. A part of the hierarchy is shown in the graph. In total, there are 20 classes on the first level and 111 leaf nodes. Some parent nodes don't have any child nodes.
I would like to fit a multi-class classifier for every parent node (so considering only the part of the hierarchy shown above, that would be 3 classifiers, one for the first level and two for the parent nodes that have children), which is called the “Local classifier per parent node approach” according to Silla & Freitas, 2011. How I understand it, the classifier for the first level is trained using the whole training set, while the classifiers for the second level are trained only using examples with labels that belong in these classes (so the classifier for the children of Catering is trained with only training examples that are labeled as Catering).
I planned on using Multinomial Naive Bayes and Support Vector Machines from scikit-learn as my classification algorithms. After training, I obtain log probabilities for the test set.
What I'm unsure about is how I should combine the probabilities that I obtain with my classifiers. I thought that I could just multiply the probability of the first level with the probabilities in the second level (or add the log probabilities) to get leaf node probabilities.
My reasoning behind that is that if a mistake is made on the first level, so that the true level 1 class has a relatively low probability, then it can be still compensated by a high probability in a child node.
After obtaining all probabilities, I would have normalized the leaf node probabilities so that they add up to 1 with softmax. However, when doing that, I get very similar probabilities for every leaf node, so that the highest leaf node probability is only a tiny bit higher than the probabilities of the other leaf nodes.
That makes sense in a way because I force every second level classifier to give me probabilities for only the children of that classifier and those add up to 1. So even if my test example does not belong to the parent node in question, one of the children classes is going to get a relatively high probability.
How can I avoid the problem of leaf node probabilities that don't discriminate enough? Or is there some other error in my reasoning that led to this result?