# Tag Info

13

After performing the PCA algorithm you get the principal components, sorted by the amount of information they hold. If you keep the whole set there is no information lost. Removing them one by one and projecting them back onto the original space you can calculate the information loss. You can plot this information loss against number of principal components ...

13

PCA is actually just a rotation. Seriously, that's all: it's a clever way to spin the data around onto a new basis. This basis has properties that make it useful as a pre-processing step for several procedures. The basis is orthonormal. This is incredibly useful if your features exhibit multicolinearity (two or more features are linearly dependent): ...

10

There is an awesome library called MPLD3 that generates interactive D3 plots. This code produces an HTML interactive plot of the popular iris dataset that is compatible with Jupyter Notebook. When the paintbrush is selected, it allows you to select a subset of data to be highlighted among all of the plots. When the cross-arrow is selected, it allows you to ...

7

PCA is unsupervised method for finding the most important components. I don't see a reason why you should want add a weight. If you know what features are important, why use PCA at all? Or perform PCA on the features where you are unsure about the importance. Further, components are created in directions with highest variance and the importance is measured ...

7

It's not uncommon for someone to label it as an unsupervised technique. You can do some analysis on the eigenvectors and that help explain behavior of the data. Naturally if your transformation still has a lot of features, then this process can be pretty hard. Nevertheless it's possible thus I consider it machine learning. Edit: Since my answer was ...

6

PCA reduces dimensionality. It does not change the number of observations you have. Nor does it change the order of the data. The n-th observation in your original dataset will still be the n-th observation post-PCA. Choosing the number of components in PCA and choosing the number of clusters in K-Means are independent of each other. Both K-Means and PCA ...

6

If you're in Python, there are a couple of packages that can automatically extract hundreds or thousands of features from your timeseries, correlate them with your labels, choose the most significant, and train models for you. https://github.com/blue-yonder/tsfresh https://github.com/rtavenar/tslearn

6

Yes, absolutely. Simply split your data into two sets feature-wise, apply PCA to one of them, and then stick them back together again. How to actually perform this will vary depending on your programming language/frameworks, but it is trivially easy in python + pandas, for example.

5

There are a number of methods for this. Here's a list: You can build a Regression model and observe the p-values of the coefficients of each variable. Pearson Correlation Spearman Correlation Kendall Correlation Mutual Information RReliefF algorithm Decision trees Principal Component Analysis (which you have tried) etc. You can search for other methods ...

5

Overfitting happens when the model fits the training dataset more than it fits the underlying distribution. In a way, it models the specific sample rather than producing a more general model of the phenomena or underlying process. It can be presented using Bayesian methods. If I use Naive Bayes then I have a simple model that might not fit either the ...

5

You're on the right track. Look at calculating a few more features, both in time and frequency domain. As long as number of samples >> number of features, you aren't likely to overfit. Is there any literature on a similar problem? If so, that always provides a great starting point. Try a boosted tree classifier, like xgboost or LightGBM. They tend to be ...

5

TL;DR PCA doesn't assume the dataset to be Gaussian distributed. If you expect the PCs to be independent, then PCA might fail to live to your expectations. Assuming that the dataset is Gaussian distributed would guarantee that the PCs are independent. Long Answer PCA doesn't assume the dataset to be Gaussian distributed Most of the sources I have found (...

5

Absolutely, it is not a learning algorithm, as you do not learn anything in PCA. However, it can be used in different learning algorithms to reach a better performance in real, likes the most of the other dimension reduction methods.

5

You can not use PCA, or at least it is not recommended, for mixed data. It is best to use Factor analysis of mixed data. You are lucky that Prince is a Python package that covers all data scenarios, borrowing from its explanation: All your variables are numeric: use principal component analysis (prince.PCA) You have a contingency table: use ...

5

Deep learning does not use dimensionality reduction because deep learning itself is a useful dimensionality reduction technique. Deep learning learns a compressed, nonlinear representation of the data through the hidden layers. Since Deep Learning can learn nonlinear mappings, it is a more flexible dimensionality reduction technique than PCA which restricted ...

5

do we always have to choose principal components based on maximum variance explained? Yes. "Maximum variance explained" is closely related to the main objective as follows. Our main objective is: for a limited budget K dimensions, what information $\mbox{a}=(a_1,...,a_K)$ to keep from original data $\mbox{x}=(x_1,...,x_D)$ ($D \gg K$) in order to be ...

5

Your emphasis on using a validation set rather than the training set for selecting $k$ is a good practice and should be followed. However, we can do even better! The parameter $k$ in $\text{PCA}$ is more special than a general hyper-parameter. Because, the solution to $\text{PCA}(k)$ already exists in $\text{PCA}(K)$, for $K > k$, which is the first $k$ ...

4

Yes, through the components_ property: import numpy, seaborn, pandas, sklearn.decomposition data = numpy.random.randn(1000, 3) @ numpy.random.randn(3,3) seaborn.pairplot(pandas.DataFrame(data, columns=['x', 'y', 'z'])); sklearn.decomposition.RandomizedPCA().fit(data).components_ > array([[ 0.43929754, 0.81097276, 0.38644644], [-0.54977152, 0....

4

Update: With this sample size you almost can't find any useful insight. One of the ways to find one to one relationship is finding correlation coefficient of two random variables. Correlation is the statistical relationship between two random variables or attributes (in your case). This coefficient is a value between 1 and -1. If the value is close to 1 it ...

4

After standardizing your data you can multiply the features with weights to assign weights before the principal component analysis. Giving higher weights means the variance within the feature goes up, which makes it more important. Standardizing (mean 0 and variance 1) is important for PCA because it is looking for a new orthogonal basis where the origin ...

4

PCA will not change the order of your points. The first point will still be the first point. As for the second, this is too unclear to answer. There is no obvious relationship between the number of clusters and the number of PCs. If you use too few PCs, your data approximation is too crude. If you use too many, then you will be working with random ...

4

The question is more related to Apache Spark architecture and map reduce; there are more than one questions here, however, the central piece of your question perhaps is For example, one of the means to determine PCs of a data is to calculate covariance matrix of the features. When using HDFS based architecture for example, the original data is ...

4

Having highly correlated features is a type of redundancy in features. And yes, it effects a regression model if you are having highly correlated features. A very nice explanation is given here. PCA is a nice choice when it comes to dimensionality reduction.

4

STEP 1 When you do Regression on this dataset, what you obtain is: $\hspace{60mm}f(x_1,x_2,x_3,x_4) = ERI$ where your $x_1,x_2,x_3,x_4, ERI$ are O2 level, Cylinder Pressure, Fuel Flow,, Engine Temp, and Engine Running Indexrespecetively. In case of Multiple Regression, you will get 1 coefficient per feature (total 4 in coef_, {$w_1,w_2,w_3,w_4$}) and one ...

4

Someone correct me if I'm wrong, but the PCA process itself doesn't assume anything about the distribution of your data. The PCA algorithm is simple - find the direction of greatest variance in your data write down the direction of the vector pointing in that direction, and 'divide' the data along that direction by its variance in that direction, so the ...

4

Think of it this way: a PCA "transform" with $k$ components essentially approximates your $n$-dimensional data points by projecting them onto a $k$-dimensional linear subspace, trying not to loose too much data variance along the way. More precisely, what you are doing is representing your original points $y \in R^n$ as: $$y \approx \mu + Vx$$ where $x \... 4 Since the students know about eigenvectors and eigenvalues, I will assume that they know about Lagrangian multipliers as well. Let's start with computing the variance of data$\vec{x}$along a given direction$\vec{e}_a$. Since we only care about the direction,$\vec{e}_a$is a vector of unit length, i.e. $$\vec{e}_a^T \vec{e}_a = 1$$ So first, the ... 3 Sounds like you are on the right track. There are many ways to attack this problem. If you want to visualize the clustering, it would help to reduce the data down to two components. This could be done via PCA or manifold learning if you want to go non-linear (http://scikit-learn.org/stable/modules/manifold.html). In terms of clustering, there are many ... 3 First question: computing$\Sigma$Actually, you perform the same computation but using a matrix operation instead of a scalar operation. You might be mislead by your notation of the feature matrix$X$that you write$x$because in fact $$X = (x_j^{(i)})_{i,j} =\Big(x^{(1)} ... x^{(i)} ... x^{(n)}\Big)^T$$ that is to say,$i$-th row of the matrix$X\$ ...

3

PCA simply finds more compact ways of representing correlated data. PCA does not explicitly compact the data in order to better explain the target variable. In some cases, most of your inputs might be correlated with each other but have minimal relevance to your target variable. That's probably what is happening in your case. Consider a toy example. Lets ...

Only top voted, non community-wiki answers of a minimum length are eligible