1
$\begingroup$

We have 4000 features and we are applying Principal Component Analysis to reduce them a small number of features from 20 to 100.

We are performing linear regression.

Both training and validation errors are worse than without any use of PCA.

However we notice that the training error is not lower and actually has become slightly larger than the validation error.

Is this something expected?

Edit:

The variance ratio of the first principal components is high enough in comparison to the others.

What we are noticing is that if you use all of the 4000 principal components then the gap between training error and validation is larger and as you repeat the process and you reduce the principal components to 1000, 200, 100, 10, etc. then the training error comes closer to the validation error (even though both are being increased as expected)

Improve this question
$\endgroup$
4
  • $\begingroup$ Why it's surprising training error is slightly larger than validation error? There's no guarantee training error is always the smallest. $\endgroup$
    – SmallChess
    Nov 4, 2016 at 4:01
  • $\begingroup$ How much variation your PCA can account for? $\endgroup$
    – SmallChess
    Nov 4, 2016 at 4:02
  • $\begingroup$ Have you checked the variance ratio across each of your component before training the model ? $\endgroup$
    – enterML
    Nov 4, 2016 at 4:30
  • $\begingroup$ I have updated the question. This might answer your questions or give you a better idea of what we are looking for $\endgroup$
    – George Pligoropoulos
    Nov 4, 2016 at 14:33

2 Answers 2

Reset to default
1
$\begingroup$

There seem to be two central issues:

  1. The errors increased after you transformed the variables using PCA - PCA creates components from the predictors, and has no relation to the target. This is important because the component describing most of the variance data (the top component of PCA) may not be a great predictor. Chances are if you are using that component, your model might be worse off and you will see an overall increase in error. I would suggest using some other method, like Random Forest or GBM, to find the variable importance of the components wrt to the predictor variable.

  2. Training error is greater than validation error - Either your model is underfitting to the data or your validation data has some easy example wrt the model. For the first case, maybe drop the regularization on your linear regression. For the second case, use cross-fold validation to get better error estimates.

Improve this answer
$\endgroup$
2
$\begingroup$

The basic principle that helps us implement PCA is it's ability to explain the variation, so it depends how much PC you are taking and how much variation they are explaining. They are definitely not explaining 100% of variation so there are good chance that Validation Error is higher.

Before applying PCA you need to understand why we apply it? We apply it to reduce computational effort and practical implementation of problem. The idea of PCA is not to give better result than actual variables in problem.

Better results are generally available with complete list of variables.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Thanks for your answer. Yes this explains it a little bit. Please take a look at the edit of my answer. This gives more detail of what is happening and I have rephrased the question. We are trying to understand why this is happening. $\endgroup$
    – George Pligoropoulos
    Nov 4, 2016 at 12:34
  • 1
    $\begingroup$ The edited part is not sufficient to answer the question and it depends on case to case actually. This all depends on how your data is placed, this scenario can change for different data. $\endgroup$
    – Rahul Sharma
    Nov 8, 2016 at 7:37
  • $\begingroup$ Hi George, if you are satisfied with answer please up vote and mark it as correct. thanks $\endgroup$
    – Rahul Sharma
    Nov 10, 2016 at 6:09
  • $\begingroup$ it seems that it does not add anything related to the question. it gives some answer but I dont know. Is this the only answer? We just take defacto that when fewer principal components are included the model will perform mostly the same on the training and validation set? $\endgroup$
    – George Pligoropoulos
    Nov 11, 2016 at 16:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.