4
$\begingroup$

In short: How to think about the type and distribution of my variables when choosing parameter values for xgboost?

Context: I have a dataset which I want to classify using the binary:logistic objective (I am using the R implementation). I wonder if visually inspecting variables in my dataset could inform selection of parameters values, and if so then how does one generally go about it.

Example: A dataset with a binary outcome variable and a mixture of binary/continuous predictors.

Outcome variable (one value per observation):

Outcome variable

A set of predictors called flavours (~150 variables) and textures (~80 variables) have binary values 0, 1. Each observation has one or multiple flavours and textures (number of f/t per observation on the horizontal axis):

Histogram number of features

Some flavours/textures occur much more likely than others (frequency on horizontal axis):

Histogram feature frequency

I added some quantitative variables to summarise flavour/texture for each observation. I calculated a frequency for each f/t (= what is the percentage of observations where this f/t occurs) and then calculated min (most unique f/t in the given observation) max (most common f/t) and mean (average frequency).

Boxplot frequency

Apart from that, there are three more continuous variables:

Boxplot additional features

What conclusions regarding model parameters one might draw from this, if any?

$\endgroup$

1 Answer 1

1
$\begingroup$

A tip: Dont

A trick:Dont

The reason? Machine learning scientific methodology is based on cross-validation. Almost all papers (and i put the almost because of yes) select everything based on cross-validation and not in previous knowledge.

Xgboost is particularly more complicated because it has a lot of math involved.

For a simpler case, lets say that you have a problem with 3 features and you want to use a Lasso regression. Which hyperparameter will you choose? Even in this really simple case you don't know. You will need to do CV and select them there.

For xgboost the only tip that I can provide based on my experience is that the only important hyperparameter is the number of trees on the ensembling. The rest seems not to be so important.

My recommendation: only tune number of trees in the xgboost and select the best in cross-validation. Unless you want to find the absolute minime, that then, you are going to explore the whooooole hyperspace. Good luck!

$\endgroup$
4
  • $\begingroup$ +1 for the answer to the question (tune hyperparameters; EDA won't provide much information on optimal hyperparameters). However, concerning which hyperparameters to search: my experience is that at least one of the many tree-complexity parameters is important, as is learning rate (though that's mostly smaller-is-better-but-requires-more-trees). Anyway, for that sort of thing there are other questions around here. $\endgroup$
    – Ben Reiniger
    Dec 31, 2020 at 15:41
  • 1
    $\begingroup$ I did not believe I would get an answer in 2020 but you made it! :) Thank you for this insight, and all the best in 2021! $\endgroup$
    – jakub
    Dec 31, 2020 at 17:56
  • $\begingroup$ @BenReiniger you can be right. Probably I got used to the catboost implementation where learning rate is automatically changed and I only end up changing is iterations. catboost.ai/docs/concepts/speed-up-training.html $\endgroup$ Dec 31, 2020 at 18:21
  • $\begingroup$ @jakub just to finish the year!! All the best for 2021! $\endgroup$ Dec 31, 2020 at 18:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.