Recently I have been using LightGBM as regressor in order to predict, on a dataset of 20 thousand observations and 40 variables.
I have two modes, 1) Production and 2) Testing. The first one just trains a model with the whole dataset. The second does the same with an 80% of the dataset and tests over the remaining 20% (80-20 done with train_test_split, from sklearn.model_selection, no seed used).
In both cases I show the neg_mean_squared_error at the end of the execution. In the first case for the whole dataset, in the second I get two values (training and testing)
I am shocked to see that while in the first case my error is around -10, in the second the values are -5 (training) and -5.3 (testing). An average value of my objective variable can be 80. I would expect to lose accuracy since I train with less data and then I test over a different sample.
Question: There is any theoretical reason that explains that a 80-20 test gets a "better" neg_mean_squared_error than in the full data case? Or it has to be a (sneaky) bug in my code?