Skip to main content
Data Science

Return to Answer

deleted 2 characters in body
Source Link
Peteris
Peteris
  • 385
  • 2
  • 6

Large number of parameters compared to data points

In general, one aspect of overfitting is trying to "invent information out of knowthing"nothing" when you want to determine a comparably large number of parameters from a limited amount of actual evidence data points.

For a simple linear regression y = ax + b there are two parameters, so for most sets of data it would be underparametrised, not overparametrised. However, let's look at the (degenerate) case of only two data points. In that situation you can always find a perfect linear regression solution - however, is that solution necessarily meaningful? Possibly not. If you treat the linear regression of two data points as a sufficient solution, that would be a prime example of overfitting.

Here is a nice example of overfitting with a linear regression by Randall Munroe of xkcd fame that illustrates this issue:

Extrapolating

Large number of parameters compared to data points

In general, one aspect of overfitting is trying to "invent information out of knowthing" when you want to determine a comparably large number of parameters from a limited amount of actual evidence data points.

For a simple linear regression y = ax + b there are two parameters, so for most sets of data it would be underparametrised, not overparametrised. However, let's look at the (degenerate) case of only two data points. In that situation you can always find a perfect linear regression solution - however, is that solution necessarily meaningful? Possibly not. If you treat the linear regression of two data points as a sufficient solution, that would be a prime example of overfitting.

Here is a nice example of overfitting with a linear regression by Randall Munroe of xkcd fame that illustrates this issue:

Extrapolating

Large number of parameters compared to data points

In general, one aspect of overfitting is trying to "invent information out of nothing" when you want to determine a comparably large number of parameters from a limited amount of actual evidence data points.

For a simple linear regression y = ax + b there are two parameters, so for most sets of data it would be underparametrised, not overparametrised. However, let's look at the (degenerate) case of only two data points. In that situation you can always find a perfect linear regression solution - however, is that solution necessarily meaningful? Possibly not. If you treat the linear regression of two data points as a sufficient solution, that would be a prime example of overfitting.

Here is a nice example of overfitting with a linear regression by Randall Munroe of xkcd fame that illustrates this issue:

Extrapolating

Source Link
Peteris
Peteris
  • 385
  • 2
  • 6

Large number of parameters compared to data points

In general, one aspect of overfitting is trying to "invent information out of knowthing" when you want to determine a comparably large number of parameters from a limited amount of actual evidence data points.

For a simple linear regression y = ax + b there are two parameters, so for most sets of data it would be underparametrised, not overparametrised. However, let's look at the (degenerate) case of only two data points. In that situation you can always find a perfect linear regression solution - however, is that solution necessarily meaningful? Possibly not. If you treat the linear regression of two data points as a sufficient solution, that would be a prime example of overfitting.

Here is a nice example of overfitting with a linear regression by Randall Munroe of xkcd fame that illustrates this issue:

Extrapolating