I am working with an ordinal classification problem with six ordered classes and I want to compare a neural network classifier with a baseline classifier that is as simple and parameter-free as possible.
In my case, I want the baseline to use only the most important single feature $X$ that the neural network uses. For this particular problem, it makes sense to look for a set of class thresholds $\{t_i\}$ that lets me classify as:
X <= t_0 -> class 0
t_0 < X <= t_1 -> class 1
t_1 < X <= t_2 -> class 2
t_2 < X <= t_3 -> class 3
t_3 < X <= t_4 -> class 4
t_4 < X -> class 5
I have looked at a few options, and decision trees seem to fit the bill. They are very simple, and do not require choosing any free parameters (unlike e.g. $k$-nearest neighbours which requires choosing a value for $k$). If I create the tree with the argument max_leaf_nodes=6
it is quite easy to extract the thresholds $\{t_i\}$ from the resulting decision tree after fitting. (If there is another method I could use that would fit the bill as well or better to achieve this, please let me know in the comments!)
I have divided my data into six folds while ensuring that the class distribution is very similar in all folds. For this baseline I balance the classes by oversampling, as the classes are originally somewhat unbalanced. (The ratio between samples in the most common class and the rarest class is around 5:1).
For five out of my six folds, the decision tree method works very well when I use that fold for testing and the rest for training. However, for the last combination of folds used for training, I get a decision tree where class 2 is not represented in the output. Instead, two of the leaf nodes represent class 1:
Is there any way to force the decision tree to build itself in such a way that the six leaf nodes of the tree represent the six different classes?