LightGBM gives different results (metrics) depending on the columns order

Question

I have two nearly identical datasets A and B which differ only in terms of columns ordering. I then train a LightGBM model on each of the two datasets with the following steps:

1. Divide each dataset into training and testing (use the same random seed and ratio for both A and B)
2. Leave the hyperparameters as pretty much defaults
3. Set random_state to a fixed number (for reproduction)
4. Tune the learning_rate using a Grid Search
5. Train a LightGBM model on the training set and test it on the testing set
6. Learning rate with the best performance on the testing set will be chosen

The output of the two models based on these two datasets is very different, which makes me think that the ordering of columns affects the performance of LightGBM models.

Do you know why this can be the case?

Answer 1

A possible explanation is this:

When the order of the columns differ, there is a little difference in the procedure.

What LightGBM, XGBoost, CatBoost, amongst other do is to select different columns from the features in your dataset in every step in the training.

The selections of these columns is done randomly: Let's say your dataset has 20 columns. The root node selects the features 1st, 3rd and 18th, on both datasets the 1st, 3rd and 18th features are different in both possible datasets. This is repeatedly done and in every step there is randomness affecting your ultimate result.

Answer 2

While the ordering of data is inconsequential in theory, it is important in practice. Considering you took steps to ensure reproducibility, Different ordering of data will alter your train-test split logic(unless you know for certain that the train sets and test sets in both cases are exactly the same). Though you don’t specify how you split the data it is highly possible that a certain assortment of data points makes the machine more robust to outliers and therefore offering better model performance. In the case that the train and test data is the same in both cases, you’d likely have to see if there is a seed/reproducibility measure (in any part of your code) that you have not taken.

Answer 3

Control random seed doesn't help generate the same results, even if the two datasets are essentially the same. I guess it is related to how LightGBM splits a tree. Random seed only ensures that for each split, it will always select features from certain column indexes. In this sense, even if two datasets are the same with different column order, since the same column indexes could refer to different features between two datasets, the selected features at a split of a tree may be different (in LightGBM) for the two datasets. This will lead to different results.

Hopefully, this issue can be fixed in LightGBM in future.

Answer 4

I can confirm that column sorting order in the training set affects LightGBM regression models metrics... And it is not a beneficial feature, because putting more important features (sorted by SHAP importance) at the "top" (or rather on the left side of pandas DataFrame) does not improve the model from alphabetical sorting, in fact the opposite. I will try to create a reproducible example and file a bug report later tonight.