0
$\begingroup$

I read through document of lightgbm,it just tells this parameter, but didn't give much explanation for it.

The explanation:

used only in dart

set this to true, if you want to use uniform drop

I knew the meaning of dart and the drop in dart,just don't know what does uniform drop means here.

And the parameter regarding category is also not clear to me,like min_data_per_group

Any explanation or the link or paper is welcomed.

$\endgroup$

1 Answer 1

2
$\begingroup$

From the xgboost documentation https://xgboost.readthedocs.io/en/stable/tutorials/dart.html.

Vinayak and Gilad-Bachrach proposed a new method to add dropout techniques from the deep neural net community to boosted trees, and reported better results in some situations.

  • Drop trees in order to solve the over-fitting.
    • Trivial trees (to correct trivial errors) may be prevented.

This argument falls in the sampling category:

  • sample_type: type of sampling algorithm
    • uniform: (default) dropped trees are selected uniformly.
    • weighted: dropped trees are selected in proportion to weight.

In essence this is lightGBMs attempt at dropout (regularization). In a NN dropout regularization randomly omits units from the hidden layers during training.

$\endgroup$
3
  • $\begingroup$ So is that means, in uniform case, we always drop 10 trees, in weighted case, it depends how many trees in total? if we have 100 trees, we drop 10, if we have 500 rounds, we drop 50? $\endgroup$ Commented Dec 6, 2023 at 11:15
  • $\begingroup$ @cloudscomputes No, this argument does not determine how many trees to drop, but rather which trees to drop. Uniform selects these trees at random. The dropout rate is in the rate_drop argument (which is a proportion). $\endgroup$ Commented Dec 6, 2023 at 11:18
  • $\begingroup$ Ok,so in uniform the chance for each tree to be overlooked is the same, while for weighted there may be difference. $\endgroup$ Commented Dec 7, 2023 at 5:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.