Combining results from classifiers trained on different test/train splits results in higher accuracy

Question

I have developed a classifier model using LightGBM. The accuracy of the model varies significantly because of the test_train_split state(between 83% and 91%). This is normal due to the nature of the data.

I have noticed that using 3 models - trained and tested on the same data, but using different random states - and combining their results gives a higher overall accuracy.

For example: Model 1 will classify Sample 1 as Category A. This result would be wrong.

However, running the three models (trained on different subsets of the data), will give the following results:
Model 1: Category A,
Model 2: Category B,
Model 3: Category B.
Category B is indeed correct.

Is this a good practice? It seems quite untraditional, but it is working. Is there a better way to get the same results? I have thought about using all data to train the model and not splitting it, however LightGBM needs a test set as far as I understand.

Answer 1

A number of observations as per your use case:

1. Please use a k-fold validation scheme, for more reliable numbers of accuracy.

2. The model or data need some more hyper-parameter tuning since random_state is majorly meant for reproducibility during development. It won't come to your model's resuce on live production.

3. You mentioned that all three models are trained and tested on same data, but later on you mentioned the three models are trained on different subsets of data, kindly clear this anomaly.

4. Model ensembling is widely used but that is majorly driven from combinations of models with different sort of arch or parameters, however training on different data and just differing in random state is not a good practice.

5. It is vital to have data in multiple independent buckets (for training/validation/testing). Training on all the data will defintely give you better results in accuracy but will fail miserably in production. It is referred to as overfitting.