# XGBoost becomes unstable when predicting more than ~300 classes

I'm using the Python implementation of XGBoosts (version 0.80) XGBoostClassifier to predict one of a large number of classes.

My feature data consists of a sparse boolean matrix of ~10M rows, ~5k columns, with a density of 0.003. (so about 15 values per row).

My targets consist of 2000+ different classes in a long-tail skewed distribution (~300 classes occur more than 10k times).

For now I'm grouping all of my uncommon classes (<10k) into 'other' so they at least count as negative examples. But ideally, I'd like to be able to use a lot more of the different classes as actual targets.

However, whenever I use, say, 800 different targets (each occurs ~5k+ times), the model becomes unstable: training takes a lot longer, the training loss doesn't seem to converge (I haven't saved the output of this unfortunately) and the predictive quality becomes increasingly poor (predict_proba returning 0.99+ for the wrong classes).

Should I just change some parameter I've overlooked? Am I running out of float precision for my loss? Is it due to sampling in the approx method? Does anyone have any ideas?

My specific classifier:

clf = XGBClassifier(
n_jobs=8,
max_depth=7, #arbitrary
min_child_weight=5, #arbitrary
subsample=0.5, #arbitrary
colsample_bytree=0.9, #arbitrary
tree_method='approx',
objective='multi:softprob',
eval_metric='mlogloss',
silent=False
)


I am affraid that it may not be a problem about tuning the XGB classifier, but about your dataset.

Even with 10M rows, a density of 0.003 over 5000 rows seems far from enough to get interesting results about 300+ classes.

I would suggest to start with a way simpler approach to confirm that the modelling approach is okay. That would mean :

• Group your variables significantly more (<100), identify and keep the one that appears significant predictors
• Group your classes significantly more (<10)
• Start with a simpler model (random forest maybe)

Once you do that and get baseline performance metrics, you can try to add more classes / variables / complexity to your model and see if it actually improve.

A one step calibration process, where you dump all your data into a very complex model is rarely the way to go.