Recently I have been using LightGBM as regressor in order to predict, on a dataset of 20 thousand observations and 40 variables.

I have two modes, 1) Production and 2) Testing. The first one just trains a model with the whole dataset. The second does the same with an 80% of the dataset and tests over the remaining 20% (80-20 done with train_test_split, from sklearn.model_selection, no seed used).

In both cases I show the neg_mean_squared_error at the end of the execution. In the first case for the whole dataset, in the second I get two values (training and testing)

I am shocked to see that while in the first case my error is around -10, in the second the values are -5 (training) and -5.3 (testing). An average value of my objective variable can be 80. I would expect to lose accuracy since I train with less data and then I test over a different sample.

Question: There is any theoretical reason that explains that a 80-20 test gets a "better" neg_mean_squared_error than in the full data case? Or it has to be a (sneaky) bug in my code?

  • $\begingroup$ -10 and -5 are not very meaningful to people in this forum without some context. you should add the general magnitude of your output. also how big is your dataset and how do you select the 80/20 split? what if you draw another 80/20 split? do you get the same result? $\endgroup$
    – oW_
    Feb 28 '19 at 23:49
  • $\begingroup$ Updated with additional info $\endgroup$
    – Ripstein
    Mar 1 '19 at 14:15

I cannot think about any theoretical reason. However:

  • It may be the case that the neg_mean_squared_error is not correctly implemented. Try another metric, like Median or Mean Percentage Error and see if that continues to happen.
  • It may be related to how the data is distributed, but still that's not common.

Lastly, I wouldn't put a model in production (if by that you mean deploying into a product, for example) without testing its performance in a test set, even if in another instance of the model, trained using cross validation, has good performance. One reason: it may be the case that a small part of the training data breaks your model, and does not generalise.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.