I have a little confusion.
What follows is from Introduction to Statistical Learning (2013) by Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani.
My understanding of what is going on is the following. The black curve is a function, let us say $y=f(x)$.
We have a random variable $g$ which I write as $g=f+\varepsilon$, $f$ plus noise.
The data points are a subset of the plane, $n$ trials of $g$, the (possibly multi)-set $T:=\{(x_i,g(x_i)):i=1,\dots,n\}$.
I imagine (incorrectly as I know that splines are not polynomials) that the yellow curve is a degree one polynomial (flexibility two) that is the best LS fit to the the trials (the training), the blue is the best degree five polynomial (flexibility six), and the green is the best degree $(20+\alpha)$ polynomial (flexibility $20+\alpha+1$).
In my head, the training should be the data points $T$, while the test should be $f$ (as in $f$ is the expectation of the test data).
I understand that the grey line is telling me that increasing the flexibility (degree of the polynomials in my head), allows me to approximate better the set $T$. However, if I have duplicates of $x$ in $T$, say $x_j=x_k=x^*$, with different $g$ values (e.g. something like $(20,3),\,(20,5)$ both in $T$), then I cannot have a polynomial (or indeed spline or any function) $p$ that has $p(x_j)$ and $p(x_k)$ different: $p(x_j)=p(x_k)=p(x^*)$, single-valued.
Therefore, if I have such duplications in the $x$ variable, I cannot reduce the MSE to zero.
In turn, the red line shows, that when we overfit the data with too much flexibility, the fitted curve is (my usage) biased to the training and so will not model well $f$, and so we have this increasing.
The problem I can't square (excuse the pun), is the dashed line.
It says minimum possible test MSE over all models. Whether 'test' refers to $f$ or $T$ this does not make sense to me.
If 'test' here means $f$, well surely this is zero? We can approximate $f$ arbitrarily well with a polynomial of large enough degree.
If 'test' here means the data $T$, we must conclude that $T$ contains $x$ duplicates: otherwise we could fit a polynomial of degree $n+1$ through all the test points and get this to zero. Therefore there must be duplicates, and so, perhaps, this theoretically best fit goes through all the points which are not duplicated, and goes through the average $g(x_i)$ of the duplicated points... and the answer turns out to be one... but then the grey line should not go below this...
Therefore I conclude that the dashed line is the best possible fit to $f$... but why isn't this zero?
Questions:
Am I right to be confused by this? Is the black $f$ the test or the training?
Am I misunderstanding something else? Perhaps these (smoothing) splines cannot well-approximate as well as polynomials?