1
$\begingroup$

I have a data set of 11 variables with allot of observations for each one. I want to make linear regression on the variables with the observed $\vec{y}=\alpha +\beta*\vec{X}$ when X is matrix. I'm trying to reduce my parameters so I activate pca algorithm on X. I get the "loading" data but i don't understand how to use it to get only four (for example) variables to estimate instead of 11.

somebody can help?

Improve this question
$\endgroup$
3
  • $\begingroup$ Welcome to the site! If I understand your question correctly, you are trying to ask how are the values helpful in prediction using Linear Regression ? I'm I right? $\endgroup$
    – Toros91
    Commented Mar 7, 2018 at 3:57
  • $\begingroup$ Not sure about your question, but a "scree plot" may the one you need to look at! $\endgroup$
    – Vasim
    Commented Mar 7, 2018 at 7:18
  • $\begingroup$ @Toros91 you are right! that's what i'm trying to do. $\endgroup$
    – אבנר יעקב
    Commented Mar 7, 2018 at 7:44

2 Answers 2

Reset to default
0
$\begingroup$

Welcome to the site!

So, the outcome which you get from PCA explain the most of your original dataset. You need to name them based on your business understanding(Assuming that you know about data, as you mentioned that you wanted to apply, Linear Regression) else you might need some Subject Matter Experts expertise.

Of-course, the Features won't be same with the original data or else what is the point in performing PCA(I know that you understood this part). To decide on the number of features, you need to look at Scree Plot.

PCA is a Dimensionality Reduction algorithm which helps you to derive new features based on the existing ones. PCA is an Unsupervised Learning Method, used when the has many features, when you don't understand anything about the data, no data dictionary etc.For better understanding on PCA you can go through this link-1,link-2.

Now before performing Linear Regression, you need to check if these new features are explaining the Target Variable by applying Predictor Importance test(PI Test), you can go through the Feature Selection test in the python,R.

Based on the outcome of PI Test you can go ahead and use those important feature for modeling and discarding the features which are not explaining the target variable well.

Finally, you can achieve the results which you are looking for.

Let me know if you are stuck somewhere.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks! Can you explain Little more on the "PI Test" and how it's done? $\endgroup$
    – אבנר יעקב
    Commented Mar 7, 2018 at 18:41
  • $\begingroup$ @אבנריעקב: appended the links, have a look and let me know if you need anything else $\endgroup$
    – Toros91
    Commented Mar 8, 2018 at 1:02
  • $\begingroup$ if you are stuck somewhere, let me know would help you. If you got what you are looking for, you can accept the answer $\endgroup$
    – Toros91
    Commented Mar 12, 2018 at 7:32
1
$\begingroup$

In general, I would suggest to use a regularization technique for reducing the dimensionality ofa data set in linear regression cases. Please refer to L1 regularization.

If you want to decrease the number variables using PCA, you should look at the lambda values that describe the variations in the principle components, then, select the a few components with the largest corresponding lambda values (eg the first four).

Note:

  1. Do a scaling if necessary.
  2. Sometimes the very first component is not very relevant and can be eliminated.
Improve this answer
$\endgroup$
2
  • $\begingroup$ that i understand, but what is exactly my four new variables? are they the first four columns of the "score"? because i get something really different from the originals observations. $\endgroup$
    – אבנר יעקב
    Commented Mar 6, 2018 at 22:16
  • $\begingroup$ Not necessarily the first four columns of scores unless they are already sorted (descendent) based on the eigenvalues (that shows the variance in the new columns). Therefore, the four columns that you select should preserve most of the variations in the dataset. It's normal to see very different from the original observation as PCA tries to project the points to a new set of dimensions where the columns are orthogonal (linearly uncorrelated). Since your original dimensions may not be necessarily orthogonal, the input and output of the PCA may look very different. $\endgroup$
    – Borhan Kazimipour
    Commented Mar 6, 2018 at 22:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.