ground truth fit is worse than cross validated fit on noisy data?

Question

I am having these weird results when playing around with cross-validation that I would greatly appreciate having any comments.

Briefly, I have a lower mean squared error (MSE) when doing regression (least-squares) using cross-validation (CV), than when using the "ground truth weights" that I used to generate the data.

Note, however, that I compute the MSE on noisy data (generated data + noise), so MSE of 0 would not be expected for noise levels above 0.

Weirdly, for high noise conditions, I get lower MSE with cross-validated least squares than with the "ground" truth weights used to generate the clean data - to which I then add different levels of noise to the input (X). Instead, if I add gaussian noise to the output (y) the "ground truth weights" perform better.

More details are below.

Simulation of data

I am generating beta from a gaussian and X from a uniform distribution. I then compute the to-be-regressed y as y = beta * X. python 3 code:

def generate_data(noise_frac):
  X = np.random.rand(ntrials,nneurons)
  X = np.random.normal(size=(ntrials,nneurons))
  
  beta = np.random.randn(nneurons)
  y = X @ beta

  # not very important how I generated noise here
  noise_x = np.random.multivariate_normal(mean=zeros(nneurons), cov=diag(np.random.rand(nneurons)), size=ntrials)

                            
  X_noise = X + noise_x*noise_frac

  return X_noise, y, beta
  
  

As you can see I also add noise to X.

Regression

I then project this noised data X_noise for different values of noise onto beta:

y_hat = (X_noise) @ beta

And compute the MSE:

mse = mean((y_hat - y)**2)

As expected, MSE increases with noise (blue line in the figure).

However, I get lower MSE if I use cross-validated least-squares! This is now an orange line in the figure.

To do CV, I split X_noise in random 100 train and test sets. In broad terms, This is how I do CV in python:

beta_lsq = pinv(X_train) @ y_train
y_hat_lsq = (X_test) @ beta_lsq
mse = mean((y_hat_lsq - y_test)**2)

On the other hand, if I add noise to y, instead of X, then everything makes sense:

P.S.: This is a crosspost from stack overflow

Answer 1

The main problem in your test is that your X has the same scale as your noise level (0-1), as a result, adding a noise changes your data distribution significantly.

This is your data distribution before and after adding noise.

It is like the noise is 50-200% more than your initial data. That's why you get a better result with CV than the "ground truth weights". Because the new fitted model by CV is tuned for an X with a new distribution.

However, if you increase your X values scale, for example, 100*np.random.rand(ntrials,nneurons), your data have the same distribution before and after adding noise.

Now if you run your code, you will get what you expected.

Answer 2

It seems that the reason why you're getting the better result has nothing to do with cross-validation, but rather with weight adjustment happening during calculation of cross-validation result in train_decode function. To check this, I have used the following simulation,

results = {"beta": [], "folds": []}
noises =  np.arange(0,1,0.1)

for noise_frac in noises:
  print(noise_frac) 
  acc_folds = []
  acc_bests = []
  for _ in  range(1000):
    X, y, beta = generate_data(noise_frac)
    beta_prime = train_decode(X,y)
    acc_folds.append(test_decoder(beta_prime,X,y))
    acc_bests.append(test_decoder(beta,X,y))

  results["beta"].append(acc_bests)
  results["folds"].append(acc_folds)

as well as before, the errors follow the same trend, and have roughly the same values,

here, the beta-result is calculated as following:

$y=\beta\times X$

$y^\prime=\beta\times (X+noise)$

$MSE = mean((y^\prime-y)^2)$

while, the would-be-cross-validation-result is calculated using adjusted $\beta$:

$y=\beta\times X$

$\beta^\prime=(X+noise)^{-1}\times y$

$y^\prime=\beta^\prime\times (X+noise)$

$MSE = mean((y^\prime-y)^2)$

To conclude all, I can say the simulation shows that adjusted weights(beta) bring the error in target values closer to irreducible error.