2
$\begingroup$

I'm studying the difference between GLM models (OLS, Logistic Regression, Zero Inflated, etc.), which are deterministic, since we can infer the parameters exactly, and some CART models (Random Forest, LightGBM, CatBoost, etc.) that are based on stochastic prediction.

What I've heard is that for stochastic models we should split into train and test to avoid over-fitting, fact that does not happen in deterministic models, because they use Linear Programming for finding the best parameters.

I've like to start some discussion about it.

My opinion is that it's true. Deterministic models are just equations solved, and it should not over-fit the data at all, and it differs from stochastic models based on randomness to make predictions.

But what I found was every course saying to split every datasets, independent if its deterministic or not.

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ Logistic and others cannot overfit? The OP looks at the solving mechanism. I can certainly specify a logistic model incorrectly and have it overfit. Also, all of the other models are not always stochastic. I can build GBMs non-stochastically. $\endgroup$
    – Craig
    Sep 23, 2021 at 11:29
  • $\begingroup$ A model is never perfect, a nearly perfect model trained on training data is overfitting. It it quite impossible to have a perfect linear regression going throug all your data points. $\endgroup$
    – Malo
    Sep 25, 2021 at 17:04

3 Answers 3

Reset to default
2
$\begingroup$

The point you are missing is: how do you know that a model performing well on your data set generalizes? The only possibility you have is to test your model on unseen data. That is why you should split your data set into training and test set.

What you don't need is a validation set in this case. Because there are no hyper-parameters to optimize there is no need for it.

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ Model specification is an issue though. How do I know I specified my model correctly (or at least close to correctly) if I do not have a validation set? If I want a data set under lock and key that I do not look at until I am ready to go into prod that lets me see the view of how the model may perform, how do I check model specification from training? All problems may not need it, but certainly many do. The problems I work on need it. I make 3 sets - independent of the algorithm choice. $\endgroup$
    – Craig
    Sep 23, 2021 at 11:50
  • 1
    $\begingroup$ You are right! Excellent reason for a validation data set. It also allows model comparison. $\endgroup$
    – Alessandro
    Sep 24, 2021 at 12:12
2
$\begingroup$

A simple counter-example: apply linear regression on a training set made of two points. By construction the linear model fits the training data perfectly. However it's unlikely to fit any realistic test set perfectly, in fact it would usually not fit the test set well at all.

This model would overfit and the only way to evaluate it would be to apply it to a test set.

Improve this answer
$\endgroup$
1
$\begingroup$

You should always split your data into train and test sets. Whether the model is deterministic or not has no relevance; all models can overfit. You can overfit a Logistic Regression if you give it enough features. Also Decision Trees are generally deterministic and are notorious for overfitting.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.