I searched this question and the answer I got was about a general regression model, rather than a single variable, linear regression model. If you increase the number of variables, you could fit a curve, instead of line to data points which were supposed to lie on a line, but what if my model is just a straight line, fitting a 100 samples ?
A simple y = mX + c model.
Is it possible to overfit this model ?
Not really.
Overfitting / underfitting is about the balance between model capacity and the complexity of data. In the case of linear regression, the model capacity is fixed - the number of parameters in the model is always equal to input dimension + 1.
So when your data is only one-dimensional, it's really about how good your data is. The only scenario where the model can be considered as overfit is when the data is degenerated - they are all equal or equivalently very close to each other, in which case the "proper fitting" model is just a horizontal line y=b. On the other hand if the data comes from a non-trivial distribution, linear regression is almost surely going to be underfitting.
It is true that a straightforward linear regression model can be overfit, even with just a straight line fitting on 100 samples. When the model has too many parameters or is too flexible in comparison to the number of data points, overfitting can happen. Overfitting can be avoided using strategies like cross-validation and regularisation. While regularisation adds a penalty term to the model's objective function to lessen overfitting, cross-validation involves dividing the data into training and testing sets.
