0
$\begingroup$

What are the parameters that add randomness to the training of a lightgbm model? (for a large dataset) I have tried setting all parameters as default and letting bin_construct_sample_cnt be greater than the data size. But the models I trained are still different for different random seeds. It is also worth noting that I have confirmed colsample_bytree is 1.0 and bagging_freq is 0, which means there are no sampling of data or feature. What can be the other factors adding randomness to the training process that I did not notice?

$\endgroup$
3
  • $\begingroup$ Hi @kimo, welcome to the site. A Google search for "lightgbm non deterministic" shows potentially relevant matches. Have you checked them to see if they apply to your case? $\endgroup$
    – noe
    Commented May 5, 2023 at 16:56
  • $\begingroup$ Er... the same thing that gives randomness to Random Forest? Bootstrap and feature subset. $\endgroup$
    – lpounng
    Commented May 8, 2023 at 4:21
  • $\begingroup$ > the same thing that gives randomness to Random Forest? That is not correct. By default, LightGBM considers all rows and all columns every time it adds a split to a tree. $\endgroup$
    – James Lamb
    Commented Nov 14, 2023 at 2:02

1 Answer 1

1
$\begingroup$

What can be the other factors adding randomness to the training process that I did not notice?

By "adding randomness to the training process", I'm assuming that you mean things that could cause a different model to be produced by training runs with the following characteristics:

  • identical training code
  • identical training data
    • including identical ordering of rows and columns
  • run on the same physical machine
  • identical software
    • e.g. lightgbm, its Python dependencies, OpenMP, etc.
  • identical available physical resources
    • e.g. similar memory and CPU available to both training runs

Given that all those things hold true, here are some reasons that consecutive training runs could produce different models:

  • numerical precision differences caused by multi-threaded operations
    • explanation: floating-point multiplication is only commutative up to a certain precision... changing the order that a set of floating-point number is multiplied by each other can change the final result. There is some randomness in which order threads complete, and therefore in what order their results might be multiplied
    • solution: set deterministic=true and num_threads=1
    • note: this will make training slower
  • numerical precision differences caused by how Dataset construction is performed
    • explanation: LightGBM converts the raw training data into its preferred representation, the Dataset. It chooses how to parallelize that work (by column or by rows) by first doing a small bit of it and checking timings. Those timings and what they imply can change randomly based on what else is going on on the machine
    • solution: set deterministic=true and force_row_wise=true
  • you've asked LightGBM to do some sampling without providing a random seed
    • explanation: LightGBM can optionally sample rows (data_sample_strategy=bagging, bagging_fraction < 1.0), columns (feature_fraction < 1.0 or feature_fraction_bynode < 1.0), and splits (extra_trees=true). If you do not explicitly set a random seed, it'll choose different samples randomly on each run.
    • solution: set seed != 0 if you change the values of any of those parameters

See the following for more discusssion:

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.