8
$\begingroup$

I have two datasets A and B which are exactly the same in terms of the number of columns, name of columns and the values. The only difference is the order of those columns. I then train the LightGBM model on each of the two datasets with the following steps

  1. Divide each dataset into training and testing (use the same random seed and ratio for both A and B)
  2. Leave the hyperparameters as pretty much default
  3. Set a random state as a fixed number (for reproduction)
  4. Tune the learning_rate using a Grid Search
  5. Train a LightGBM model on the training set and test it on the testing set
  6. Learning rate with the best performance on the testing set will be chosen

The output models on the two datasets are very different, which makes me thinks that the order of columns does affect the performance of the model training using LightGBM.

Do you know why this is the case?

$\endgroup$
6
$\begingroup$

A possible explanation is this:

When the order of the columns differ, there is a little difference in the procedure.

What LightGBM, XGBoost, CatBoost, amongst other do is to select different columns from the features in your dataset in every step in the training.

The selections of these columns is done randomly: Let's say your dataset has 20 columns. The root node selects the features 1st, 3rd and 18th, on both datasets the 1st, 3rd and 18th features are different in both possible datasets. This is repeatedly done and in every step there is randomness affecting your ultimate result.

$\endgroup$
  • $\begingroup$ How can we control that randomness when the algorithm selects a subset of features to build a decision tree? That was also my only thought to answer this situation. Moreover, I guess if we always select all features per tree, the algorithm will use Gini (or something similar) to calculate the feature importance at each step, which won't create an randomness. $\endgroup$ – Duy Bui May 1 '19 at 10:22
  • $\begingroup$ lightgbm allows the user to set the random seeds used for row and column sampling. $\endgroup$ – bradS May 1 '19 at 10:47
  • 1
    $\begingroup$ @bradS: I didn't set the seed as a hyperparameter in the LightGBM but I checked again and seeds should be set as a fixed number by default. That means it should have the same result, which is not the case here. lightgbm.readthedocs.io/en/latest/Parameters.html $\endgroup$ – Duy Bui May 1 '19 at 12:40
3
$\begingroup$

While the ordering of data is inconsequential in theory, it is important in practice. Considering you took steps to ensure reproducibility, Different ordering of data will alter your train-test split logic(unless you know for certain that the train sets and test sets in both cases are exactly the same). Though you don’t specify how you split the data it is highly possible that a certain assortment of data points makes the machine more robust to outliers and therefore offering better model performance. In the case that the train and test data is the same in both cases, you’d likely have to see if there is a seed/reproducibility measure (in any part of your code) that you have not taken.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.