I am having these weird results when playing around with cross validation that I would greatly appreciate to have any comments.

Briefly, I have a lower mean squared error (MSE) when doing regression (least-squares) using cross-valitation (CV), than when using the "ground truth weights" that I used to generate the data.

Note however, that I compute the MSE on noisy data (generated data + noise), so MSE of 0 would not be expected for noise levels above 0.

Weirdly, for high noise conditions, I get lower MSE with cross validated least squares than with the "ground" truth weights used to generate the clean data - to which I then add different levels of noise to the input (X). Instead, if I add guassian noise to the output (y) the "ground truth weights" perform better.

More details below.

Simulation of data

I am generating beta from a guassian and X from a uniform distribution. I then compute the to-be-regressed y as y = beta * X. python 3 code:

def generate_data(noise_frac):
  X = np.random.rand(ntrials,nneurons)
  X = np.random.normal(size=(ntrials,nneurons))
  
  beta = np.random.randn(nneurons)
  y = X @ beta

  # not very important how I generated noise here
  noise_x = np.random.multivariate_normal(mean=zeros(nneurons), cov=diag(np.random.rand(nneurons)), size=ntrials)

                            
  X_noise = X + noise_x*noise_frac

  return X_noise, y, beta
  
  

As you can see I also add noise to X.

Regression

I then project this noised data X_noise for different values of noise onto beta:

y_hat = (X_noise) @ beta

And compute the MSE:

mse = mean((y_hat - y)**2)

As expected, MSE increases with noise (blue line in the figure).

However, I get lower MSE if I use cross validated least-squares! This is now orange line in the figure.

To do CV, I split X_noise in random 100 train and test sets. In broad terms, This is how I do CV in python:

beta_lsq = pinv(X_train) @ y_train
y_hat_lsq = (X_test) @ beta_lsq
mse = mean((y_hat_lsq - y_test)**2)

On the other hand, if I add noise to y, instead of X, then everything makes sense:

Thank you very much in advance!

PS: This is a crosspost from stack overflow

I am having these weird results when playing around with cross-validation that I would greatly appreciate having any comments.

Briefly, I have a lower mean squared error (MSE) when doing regression (least-squares) using cross-validation (CV), than when using the "ground truth weights" that I used to generate the data.

Note, however, that I compute the MSE on noisy data (generated data + noise), so MSE of 0 would not be expected for noise levels above 0.

Weirdly, for high noise conditions, I get lower MSE with cross-validated least squares than with the "ground" truth weights used to generate the clean data - to which I then add different levels of noise to the input (X). Instead, if I add gaussian noise to the output (y) the "ground truth weights" perform better.

More details are below.

Simulation of data

I am generating beta from a gaussian and X from a uniform distribution. I then compute the to-be-regressed y as y = beta * X. python 3 code:

def generate_data(noise_frac):
  X = np.random.rand(ntrials,nneurons)
  X = np.random.normal(size=(ntrials,nneurons))
  
  beta = np.random.randn(nneurons)
  y = X @ beta

  # not very important how I generated noise here
  noise_x = np.random.multivariate_normal(mean=zeros(nneurons), cov=diag(np.random.rand(nneurons)), size=ntrials)

                            
  X_noise = X + noise_x*noise_frac

  return X_noise, y, beta
  
  

As you can see I also add noise to X.

Regression

I then project this noised data X_noise for different values of noise onto beta:

y_hat = (X_noise) @ beta

And compute the MSE:

mse = mean((y_hat - y)**2)

As expected, MSE increases with noise (blue line in the figure).

However, I get lower MSE if I use cross-validated least-squares! This is now an orange line in the figure.

To do CV, I split X_noise in random 100 train and test sets. In broad terms, This is how I do CV in python:

beta_lsq = pinv(X_train) @ y_train
y_hat_lsq = (X_test) @ beta_lsq
mse = mean((y_hat_lsq - y_test)**2)

On the other hand, if I add noise to y, instead of X, then everything makes sense:

P.S.: This is a crosspost from stack overflow

I am having these weird results when playing around with cross validation that I would greatly appreciate to have any comments.

Briefly, I have a lower mean squared error (MSE) when doing regression (least-squares) using cross-validation (CV) than when using the "ground truth weights" that I used to generate the data.

Note, however, that I compute the MSE on noisy data (generated data + noise), so MSE of 0 would not be expected for noise levels above 0.

Weirdly, for high noise conditions, I get lower MSE with cross validated least squares than with the "ground" truth weights used to generate the clean data - to which I then add different levels of noise to the input (X). Instead, if I add Gaussian noise to the output (y), the "ground truth weights" perform better.

More details below:

Simulation of data

I am generating beta from a Gaussian and X from a uniform distribution. I then compute the to-be-regressed y as y = beta * X.

Python 3 code:

def generate_data(noise_frac):
  X = np.random.rand(ntrials,nneurons)
  X = np.random.normal(size=(ntrials,nneurons))
  
  beta = np.random.randn(nneurons)
  y = X @ beta

  # not very important how I generated noise here
  noise_x = np.random.multivariate_normal(mean=zeros(nneurons), cov=diag(np.random.rand(nneurons)), size=ntrials)

                            
  X_noise = X + noise_x*noise_frac

  return X_noise, y, beta
  
  

As you can see, I also add noise to X.

Regression

I then project this noised data X_noise for different values of noise onto beta:

y_hat = (X_noise) @ beta

and compute the MSE:

mse = mean((y_hat - y)**2)

As expected, MSE increases with noise (blue line in the figure).

However, I get lower MSE if I use cross validated least-squares! This is now orange line in the figure.

To do CV, I split X_noise in random 100 train and test sets. In broad terms, This is how I do CV in python:

beta_lsq = pinv(X_train) @ y_train
y_hat_lsq = (X_test) @ beta_lsq
mse = mean((y_hat_lsq - y_test)**2)

On the other hand, if I add noise to y, instead of X, then everything makes sense:

Thank you very much in advance!

PS: This is a crosspost from Stack Overflow.

I am having these weird results when playing around with cross validation that I would greatly appreciate to have any comments.

Briefly, I have a lower mean squared error (MSE) when doing regression (least-squares) using cross-valitation (CV), than when using the "ground truth weights" that I used to generate the data.

Note however, that I compute the MSE on noisy data (generated data + noise), so MSE of 0 would not be expected for noise levels above 0.

Weirdly, for high noise conditions, I get lower MSE with cross validated least squares than with the "ground" truth weights used to generate the clean data - to which I then add different levels of noise to the input (X). Instead, if I add guassian noise to the output (y) the "ground truth weights" perform better.

More details below.

Simulation of data

I am generating beta from a guassian and X from a uniform distribution. I then compute the to-be-regressed y as y = beta * X. python 3 code:

def generate_data(noise_frac):
  X = np.random.rand(ntrials,nneurons)
  X = np.random.normal(size=(ntrials,nneurons))
  
  beta = np.random.randn(nneurons)
  y = X @ beta

  # not very important how I generated noise here
  noise_x = np.random.multivariate_normal(mean=zeros(nneurons), cov=diag(np.random.rand(nneurons)), size=ntrials)

                            
  X_noise = X + noise_x*noise_frac

  return X_noise, y, beta
  
  

As you can see I also add noise to X.

Regression

I then project this noised data X_noise for different values of noise onto beta:

y_hat = (X_noise) @ beta

And compute the MSE:

mse = mean((y_hat - y)**2)

As expected, MSE increases with noise (blue line in the figure).

However, I get lower MSE if I use cross validated least-squares! This is now orange line in the figure.

To do CV, I split X_noise in random 100 train and test sets. In broad terms, This is how I do CV in python:

beta_lsq = pinv(X_train) @ y_train
y_hat_lsq = (X_test) @ beta_lsq
mse = mean((y_hat_lsq - y_test)**2)

On the other hand, if I add noise to y, instead of X, then everything makes sense:

Thank you very much in advance!

PS: This is a crosspost from stack overflow

I am having these weird results when playing around with cross validation that I would greatly appreciate to have any comments.

Briefly, I have a lower mean squared error (MSE) when doing regression (least-squares) using cross-valitation (CV), than when using the "ground truth weights" that I used to generate the data.

Note however, that I compute the MSE on noisy data (generated data + noise), so MSE of 0 would not be expected for noise levels above 0.

Weirdly, for high noise conditions, I get lower MSE with cross validated least squares than with the "ground" truth weights used to generate the clean data - to which I then add different levels of noise to the input (X). Instead, if I add guassian noise to the output (y) the "ground truth weights" perform better.

More details below.

Simulation of data

I am generating beta from a guassian and X from a uniform distribution. I then compute the to-be-regressed y as y = beta * X. python 3 code:

def generate_data(noise_frac):
  X = np.random.rand(ntrials,nneurons)
  X = np.random.normal(size=(ntrials,nneurons))
  
  beta = np.random.randn(nneurons)
  y = X @ beta

  # not very important how I generated noise here
  noise_x = np.random.multivariate_normal(mean=zeros(nneurons), cov=diag(np.random.rand(nneurons)), size=ntrials)

                            
  X_noise = X + noise_x*noise_frac

  return X_noise, y, beta
  
  

As you can see I also add noise to X.

Regression

I then project this noised data X_noise for different values of noise onto beta:

y_hat = (X_noise) @ beta

And compute the MSE:

mse = mean((y_hat - y)**2)

As expected, MSE increases with noise (blue line in the figure).

However, I get lower MSE if I use cross validated least-squares! This is now orange line in the figure.

To do CV, I split X_noise in random 100 train and test sets. In broad terms, This is how I do CV in python:

beta_lsq = pinv(X_train) @ y_train
y_hat_lsq = (X_test) @ beta_lsq
mse = mean((y_hat_lsq - y_test)**2)

On the other hand, if I add noise to y, instead of X, then everything makes sense:

Thank you very much in advance!

PS: This is a crosspost from stack overflow

I am having these weird results when playing around with cross validation that I would greatly appreciate to have any comments.

Briefly, I have a lower mean squared error (MSE) when doing regression (least-squares) using cross-validation (CV) than when using the "ground truth weights" that I used to generate the data.

Note, however, that I compute the MSE on noisy data (generated data + noise), so MSE of 0 would not be expected for noise levels above 0.

Weirdly, for high noise conditions, I get lower MSE with cross validated least squares than with the "ground" truth weights used to generate the clean data - to which I then add different levels of noise to the input (X). Instead, if I add Gaussian noise to the output (y), the "ground truth weights" perform better.

More details below:

Simulation of data

I am generating beta from a Gaussian and X from a uniform distribution. I then compute the to-be-regressed y as y = beta * X.

Python 3 code:

def generate_data(noise_frac):
  X = np.random.rand(ntrials,nneurons)
  X = np.random.normal(size=(ntrials,nneurons))
  
  beta = np.random.randn(nneurons)
  y = X @ beta

  # not very important how I generated noise here
  noise_x = np.random.multivariate_normal(mean=zeros(nneurons), cov=diag(np.random.rand(nneurons)), size=ntrials)

                            
  X_noise = X + noise_x*noise_frac

  return X_noise, y, beta
  
  

As you can see, I also add noise to X.

Regression

I then project this noised data X_noise for different values of noise onto beta:

y_hat = (X_noise) @ beta

and compute the MSE:

mse = mean((y_hat - y)**2)

As expected, MSE increases with noise (blue line in the figure).

However, I get lower MSE if I use cross validated least-squares! This is now orange line in the figure.

To do CV, I split X_noise in random 100 train and test sets. In broad terms, This is how I do CV in python:

beta_lsq = pinv(X_train) @ y_train
y_hat_lsq = (X_test) @ beta_lsq
mse = mean((y_hat_lsq - y_test)**2)

On the other hand, if I add noise to y, instead of X, then everything makes sense:

Thank you very much in advance!

PS: This is a crosspost from Stack Overflow.