We have 4000 features and we are applying Principal Component Analysis to reduce them a small number of features from 20 to 100.
We are performing linear regression.
Both training and validation errors are worse than without any use of PCA.
However we notice that the training error is not lower and actually has become slightly larger than the validation error.
Is this something expected?
The variance ratio of the first principal components is high enough in comparison to the others.
What we are noticing is that if you use all of the 4000 principal components then the gap between training error and validation is larger and as you repeat the process and you reduce the principal components to 1000, 200, 100, 10, etc. then the training error comes closer to the validation error (even though both are being increased as expected)