This is more of an open question with people which have experience in this. I'm working on a multi-class multi-label classification for chest x-rays. I would like to know how much can reducing the number of classes increase performance for remaining classes before moving forward and making big changes to the training pipeline.
For the moment, there's about 50 classes with AUC performance varying between 0.7 and <~ .99. Final outputs used are some tweaked sigmoids and loss function is based on cross entropy as it's multi class multi label. However, not all 50 classes are equally important. One option would be to give class weights. But I'm also thinking about just dropping or merging some classes instead. Hence my question:
Has anyone experience a significant increase in performance for the remaining classes by merging / dropping-out some classes ?