Minimum Possible Test MSE

Question

I have a little confusion.

What follows is from Introduction to Statistical Learning (2013) by Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani.

My understanding of what is going on is the following. The black curve is a function, let us say $y=f(x)$.

We have a random variable $g$ which I write as $g=f+\varepsilon$, $f$ plus noise.

The data points are a subset of the plane, $n$ trials of $g$, the (possibly multi)-set $T:=\{(x_i,g(x_i)):i=1,\dots,n\}$.

I imagine (incorrectly as I know that splines are not polynomials) that the yellow curve is a degree one polynomial (flexibility two) that is the best LS fit to the the trials (the training), the blue is the best degree five polynomial (flexibility six), and the green is the best degree $(20+\alpha)$ polynomial (flexibility $20+\alpha+1$).

In my head, the training should be the data points $T$, while the test should be $f$ (as in $f$ is the expectation of the test data).

I understand that the grey line is telling me that increasing the flexibility (degree of the polynomials in my head), allows me to approximate better the set $T$. However, if I have duplicates of $x$ in $T$, say $x_j=x_k=x^*$, with different $g$ values (e.g. something like $(20,3),\,(20,5)$ both in $T$), then I cannot have a polynomial (or indeed spline or any function) $p$ that has $p(x_j)$ and $p(x_k)$ different: $p(x_j)=p(x_k)=p(x^*)$, single-valued.

Therefore, if I have such duplications in the $x$ variable, I cannot reduce the MSE to zero.

In turn, the red line shows, that when we overfit the data with too much flexibility, the fitted curve is (my usage) biased to the training and so will not model well $f$, and so we have this increasing.

The problem I can't square (excuse the pun), is the dashed line.

It says minimum possible test MSE over all models. Whether 'test' refers to $f$ or $T$ this does not make sense to me.

If 'test' here means $f$, well surely this is zero? We can approximate $f$ arbitrarily well with a polynomial of large enough degree.

If 'test' here means the data $T$, we must conclude that $T$ contains $x$ duplicates: otherwise we could fit a polynomial of degree $n+1$ through all the test points and get this to zero. Therefore there must be duplicates, and so, perhaps, this theoretically best fit goes through all the points which are not duplicated, and goes through the average $g(x_i)$ of the duplicated points... and the answer turns out to be one... but then the grey line should not go below this...

Therefore I conclude that the dashed line is the best possible fit to $f$... but why isn't this zero?

Questions:

1. Am I right to be confused by this? Is the black $f$ the test or the training?

2. Am I misunderstanding something else? Perhaps these (smoothing) splines cannot well-approximate as well as polynomials?

Answer 1

Important disclaimer: I'm not a statistician and I'm not sure about my interpretation!

I also thought at first about the duplicates, but I think the problem might be with this assumption:

In my head, the training should be the data points 𝑇, while the test should be 𝑓 (as in 𝑓 is the expectation of the test data).

Specifically the last part: in principle the test set is made of points from the same distribution as the training data, with the same risk of noise. In other words, the test set $t$ is similar to $T$: $t=\{(x_i,g(x_i)):i=1,\dots,m\}$ (and not $t=\{(x_i,f(x_i)):i=1,\dots,m\}$).

If 'test' here means the data 𝑇, we must conclude that 𝑇 contains 𝑥 duplicates: otherwise we could fit a polynomial of degree 𝑛+1 through all the test points and get this to zero.

Importantly, the test set $t$ is different from the training set $T$, and the estimated function $\hat{f}$ is based only on the points in $T$. So this way it makes sense that even a perfect estimate $\hat{f}=f$ might not be able to predict the true (noisy) value for every point $x\in t$. That could explain the non-zero minimum test MSE.