1
$\begingroup$

I'm using the Python implementation of XGBoosts (version 0.80) XGBoostClassifier to predict one of a large number of classes.

My feature data consists of a sparse boolean matrix of ~10M rows, ~5k columns, with a density of 0.003. (so about 15 values per row).

My targets consist of 2000+ different classes in a long-tail skewed distribution (~300 classes occur more than 10k times).

For now I'm grouping all of my uncommon classes (<10k) into 'other' so they at least count as negative examples. But ideally, I'd like to be able to use a lot more of the different classes as actual targets.

However, whenever I use, say, 800 different targets (each occurs ~5k+ times), the model becomes unstable: training takes a lot longer, the training loss doesn't seem to converge (I haven't saved the output of this unfortunately) and the predictive quality becomes increasingly poor (predict_proba returning 0.99+ for the wrong classes).

Should I just change some parameter I've overlooked? Am I running out of float precision for my loss? Is it due to sampling in the approx method? Does anyone have any ideas?

My specific classifier:

clf = XGBClassifier(
    n_jobs=8,
    max_depth=7, #arbitrary
    min_child_weight=5, #arbitrary
    subsample=0.5, #arbitrary
    colsample_bytree=0.9, #arbitrary
    tree_method='approx',
    objective='multi:softprob',
    eval_metric='mlogloss',
    silent=False
)
$\endgroup$
0
$\begingroup$

I am affraid that it may not be a problem about tuning the XGB classifier, but about your dataset.

Even with 10M rows, a density of 0.003 over 5000 rows seems far from enough to get interesting results about 300+ classes.

I would suggest to start with a way simpler approach to confirm that the modelling approach is okay. That would mean :

  • Group your variables significantly more (<100), identify and keep the one that appears significant predictors
  • Group your classes significantly more (<10)
  • Start with a simpler model (random forest maybe)

Once you do that and get baseline performance metrics, you can try to add more classes / variables / complexity to your model and see if it actually improve.

A one step calibration process, where you dump all your data into a very complex model is rarely the way to go.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.