1
$\begingroup$

I have a little confusion.

What follows is from Introduction to Statistical Learning (2013) by Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani.

enter image description here

My understanding of what is going on is the following. The black curve is a function, let us say $y=f(x)$.

We have a random variable $g$ which I write as $g=f+\varepsilon$, $f$ plus noise.

The data points are a subset of the plane, $n$ trials of $g$, the (possibly multi)-set $T:=\{(x_i,g(x_i)):i=1,\dots,n\}$.

I imagine (incorrectly as I know that splines are not polynomials) that the yellow curve is a degree one polynomial (flexibility two) that is the best LS fit to the the trials (the training), the blue is the best degree five polynomial (flexibility six), and the green is the best degree $(20+\alpha)$ polynomial (flexibility $20+\alpha+1$).

In my head, the training should be the data points $T$, while the test should be $f$ (as in $f$ is the expectation of the test data).

I understand that the grey line is telling me that increasing the flexibility (degree of the polynomials in my head), allows me to approximate better the set $T$. However, if I have duplicates of $x$ in $T$, say $x_j=x_k=x^*$, with different $g$ values (e.g. something like $(20,3),\,(20,5)$ both in $T$), then I cannot have a polynomial (or indeed spline or any function) $p$ that has $p(x_j)$ and $p(x_k)$ different: $p(x_j)=p(x_k)=p(x^*)$, single-valued.

Therefore, if I have such duplications in the $x$ variable, I cannot reduce the MSE to zero.

In turn, the red line shows, that when we overfit the data with too much flexibility, the fitted curve is (my usage) biased to the training and so will not model well $f$, and so we have this increasing.

The problem I can't square (excuse the pun), is the dashed line.

It says minimum possible test MSE over all models. Whether 'test' refers to $f$ or $T$ this does not make sense to me.

If 'test' here means $f$, well surely this is zero? We can approximate $f$ arbitrarily well with a polynomial of large enough degree.

If 'test' here means the data $T$, we must conclude that $T$ contains $x$ duplicates: otherwise we could fit a polynomial of degree $n+1$ through all the test points and get this to zero. Therefore there must be duplicates, and so, perhaps, this theoretically best fit goes through all the points which are not duplicated, and goes through the average $g(x_i)$ of the duplicated points... and the answer turns out to be one... but then the grey line should not go below this...

Therefore I conclude that the dashed line is the best possible fit to $f$... but why isn't this zero?

Questions:

  1. Am I right to be confused by this? Is the black $f$ the test or the training?

  2. Am I misunderstanding something else? Perhaps these (smoothing) splines cannot well-approximate as well as polynomials?

$\endgroup$
1
  • 2
    $\begingroup$ Fyi for this kind of theoretical questions you might find get answers by real statisticians on stats.stackexchange.com ;) $\endgroup$
    – Erwan
    Sep 27, 2019 at 23:54

1 Answer 1

1
$\begingroup$

Important disclaimer: I'm not a statistician and I'm not sure about my interpretation!

I also thought at first about the duplicates, but I think the problem might be with this assumption:

In my head, the training should be the data points 𝑇, while the test should be 𝑓 (as in 𝑓 is the expectation of the test data).

Specifically the last part: in principle the test set is made of points from the same distribution as the training data, with the same risk of noise. In other words, the test set $t$ is similar to $T$: $t=\{(x_i,g(x_i)):i=1,\dots,m\}$ (and not $t=\{(x_i,f(x_i)):i=1,\dots,m\}$).

If 'test' here means the data 𝑇, we must conclude that 𝑇 contains 𝑥 duplicates: otherwise we could fit a polynomial of degree 𝑛+1 through all the test points and get this to zero.

Importantly, the test set $t$ is different from the training set $T$, and the estimated function $\hat{f}$ is based only on the points in $T$. So this way it makes sense that even a perfect estimate $\hat{f}=f$ might not be able to predict the true (noisy) value for every point $x\in t$. That could explain the non-zero minimum test MSE.

$\endgroup$
3
  • $\begingroup$ So perhaps the dashed line is an expectation? By chance any model could match the test data perfectly. $\endgroup$ Sep 28, 2019 at 5:15
  • $\begingroup$ @JPMcCarthy I don't think so because if one considers any model obtainable by chance then there's no point involving a training set. I think it's supposed to represent the best possible estimate based on the training data, that is f itself. It's just that f itself wouldn't obtain a 0 MSE on the test data because of the noise. $\endgroup$
    – Erwan
    Sep 28, 2019 at 10:57
  • $\begingroup$ I am saying that the expectation of the MSE of $f$ to a suitable number of trials of $g$ would be one... this matches what you are saying here. $\endgroup$ Sep 29, 2019 at 15:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.