I have two nearly identical datasets A and B which differ only in terms of columns ordering. I then train a LightGBM model on each of the two datasets with the following steps:
- Divide each dataset into training and testing (use the same random seed and ratio for both A and B)
- Leave the hyperparameters as pretty much defaults
random_stateto a fixed number (for reproduction)
- Tune the
learning_rateusing a Grid Search
- Train a LightGBM model on the training set and test it on the testing set
- Learning rate with the best performance on the testing set will be chosen
The output of the two models based on these two datasets is very different, which makes me think that the ordering of columns affects the performance of LightGBM models.
Do you know why this can be the case?