Classifier Chains

Question

My question is about creating classifier chains

The problem I am working with is a multiclass problem where I have to assign one of 7 possible classes.

I started by training a single random forest based classifier for this purpose and the results were not that great. The expectation is about 95%+ for Precision, whereas I was at 79.

So I started working with training 7 different classifiers for each class. When I take class 1, for the purpose of training, I convert the class labels of other classes to ‘Other’.

By doing this process, the performance of the individual classifiers registered ~ 95%+.

Then I thought, may be lets chain the classifiers (in a waterfall manner) in a way that the most accurate model gets first attempt at classification, the next most accurate gets left over rows to make prediction on, the third most accurate one goes after that so on and so forth. The training process was still the same(as for 7 individual classifiers).

Concretely, let us say classifier 1 predicts on 100 rows and finds 10 to be of Class 1, the other 90 will be ‘Other’. Classifier 2 will then do predictions on the remaining 90 which will be marked as ‘Other’. Classifier 3 will then go on the left over and so on and so forth.

Problem is, the results are nowhere near the when they go individually. It is down to high 70s.

I am trying to figure out what might be the reason for the steep decline in performance and wondering if this is expected or I need to modify something to achieve somewhat comparative performance.

I appreciate any input/suggestions/comments.

Answer 1

The OP reports that when a series of one-vs-rest classifiers are chained together in an ensemble from most accurate to least, the overall predictive accuracy of the ensemble decreases compared to the unchained version.

This makes perfect sense. Imagine a simpler case of 3 classes of data, A, B, & C that are used to build the chain you describe: AvsBC, BvAC, and CvAB. Let's assume the order described is in most-to-least accurate.

Now run a single instance x through this chain. Suppose classifier AvsBC assigns x a posterior probability Pr(A) = 0.51. Under this result the ensemble would presumably stop, and never explore the other options, and thus might miss out on higher posterior probability assignments (e.g., under BvAC you might get Pr(B) = 0.60). Only by running your instances through all classifiers can you maximize your accuracy.

Answer 2

Try running all of the models for each of the predictions. You will need get the probability of the prediction. Take as the classification the individual model which gives the maximum probability. Using your proposed "waterfall" does not allow the other models to have a say in the prediction of each new input. Also, are you saying that when you switched form one model with 7 classes to 7 models with 2 classes each the accuracy went from ~70 to ~95? That is interesting. Maybe you should look to see which features are the strongest predictors in each of your binary models (see variable importance).