Hot answers tagged

16

PCA is actually just a rotation. Seriously, that's all: it's a clever way to spin the data around onto a new basis. This basis has properties that make it useful as a pre-processing step for several procedures. The basis is orthonormal. This is incredibly useful if your features exhibit multicolinearity (two or more features are linearly dependent): ...


14

After performing the PCA algorithm you get the principal components, sorted by the amount of information they hold. If you keep the whole set there is no information lost. Removing them one by one and projecting them back onto the original space you can calculate the information loss. You can plot this information loss against number of principal components ...


10

There is an awesome library called MPLD3 that generates interactive D3 plots. This code produces an HTML interactive plot of the popular iris dataset that is compatible with Jupyter Notebook. When the paintbrush is selected, it allows you to select a subset of data to be highlighted among all of the plots. When the cross-arrow is selected, it allows you to ...


9

It's not uncommon for someone to label it as an unsupervised technique. You can do some analysis on the eigenvectors and that help explain behavior of the data. Naturally if your transformation still has a lot of features, then this process can be pretty hard. Nevertheless it's possible thus I consider it machine learning. Edit: Since my answer was ...


7

PCA is unsupervised method for finding the most important components. I don't see a reason why you should want add a weight. If you know what features are important, why use PCA at all? Or perform PCA on the features where you are unsure about the importance. Further, components are created in directions with highest variance and the importance is measured ...


6

PCA reduces dimensionality. It does not change the number of observations you have. Nor does it change the order of the data. The n-th observation in your original dataset will still be the n-th observation post-PCA. Choosing the number of components in PCA and choosing the number of clusters in K-Means are independent of each other. Both K-Means and PCA ...


6

Overfitting happens when the model fits the training dataset more than it fits the underlying distribution. In a way, it models the specific sample rather than producing a more general model of the phenomena or underlying process. It can be presented using Bayesian methods. If I use Naive Bayes then I have a simple model that might not fit either the ...


6

If you're in Python, there are a couple of packages that can automatically extract hundreds or thousands of features from your timeseries, correlate them with your labels, choose the most significant, and train models for you. https://github.com/blue-yonder/tsfresh https://github.com/rtavenar/tslearn


6

TL;DR PCA does assume normal distribution of features See p.55 SAS book1 or Rummel, 19702 or Mardia, 19793. If you expect the PCs to be independent, then PCA might fail to live to your expectations. Assuming that the dataset is Gaussian distributed would guarantee that the PCs are independent. Long Answer PCA doesn't assume the dataset to be Gaussian ...


6

Absolutely, it is not a learning algorithm, as you do not learn anything in PCA. However, it can be used in different learning algorithms to reach a better performance in real, likes the most of the other dimension reduction methods.


6

Yes, absolutely. Simply split your data into two sets feature-wise, apply PCA to one of them, and then stick them back together again. How to actually perform this will vary depending on your programming language/frameworks, but it is trivially easy in python + pandas, for example.


5

There are a number of methods for this. Here's a list: You can build a Regression model and observe the p-values of the coefficients of each variable. Pearson Correlation Spearman Correlation Kendall Correlation Mutual Information RReliefF algorithm Decision trees Principal Component Analysis (which you have tried) etc. You can search for other methods ...


5

The question is more related to Apache Spark architecture and map reduce; there are more than one questions here, however, the central piece of your question perhaps is For example, one of the means to determine PCs of a data is to calculate covariance matrix of the features. When using HDFS based architecture for example, the original data is distributed ...


5

You're on the right track. Look at calculating a few more features, both in time and frequency domain. As long as number of samples >> number of features, you aren't likely to overfit. Is there any literature on a similar problem? If so, that always provides a great starting point. Try a boosted tree classifier, like xgboost or LightGBM. They tend to be ...


5

You can not use PCA, or at least it is not recommended, for mixed data. It is best to use Factor analysis of mixed data. You are lucky that Prince is a Python package that covers all data scenarios, borrowing from its explanation: All your variables are numeric: use principal component analysis (prince.PCA) You have a contingency table: use ...


5

Deep learning does not use dimensionality reduction because deep learning itself is a useful dimensionality reduction technique. Deep learning learns a compressed, nonlinear representation of the data through the hidden layers. Since Deep Learning can learn nonlinear mappings, it is a more flexible dimensionality reduction technique than PCA which restricted ...


5

do we always have to choose principal components based on maximum variance explained? Yes. "Maximum variance explained" is closely related to the main objective as follows. Our main objective is: for a limited budget K dimensions, what information $\mbox{a}=(a_1,...,a_K)$ to keep from original data $\mbox{x}=(x_1,...,x_D)$ ($D \gg K$) in order to be ...


5

Your emphasis on using a validation set rather than the training set for selecting $k$ is a good practice and should be followed. However, we can do even better! The parameter $k$ in $\text{PCA}$ is more special than a general hyper-parameter. Because, the solution to $\text{PCA}(k)$ already exists in $\text{PCA}(K)$, for $K > k$, which is the first $k$ ...


5

Apparently this functionality is left out intentionally, see here. I'm afraid you have to use SVD, but that should be fairly straightforward: def pca(X): mean = X.mean(axis=0) center = X - mean _, stds, pcs = np.linalg.svd(center/np.sqrt(X.shape[0])) return stds**2, pcs


4

Yes, through the components_ property: import numpy, seaborn, pandas, sklearn.decomposition data = numpy.random.randn(1000, 3) @ numpy.random.randn(3,3) seaborn.pairplot(pandas.DataFrame(data, columns=['x', 'y', 'z'])); sklearn.decomposition.RandomizedPCA().fit(data).components_ > array([[ 0.43929754, 0.81097276, 0.38644644], [-0.54977152, 0....


4

Update: With this sample size you almost can't find any useful insight. One of the ways to find one to one relationship is finding correlation coefficient of two random variables. Correlation is the statistical relationship between two random variables or attributes (in your case). This coefficient is a value between 1 and -1. If the value is close to 1 it ...


4

After standardizing your data you can multiply the features with weights to assign weights before the principal component analysis. Giving higher weights means the variance within the feature goes up, which makes it more important. Standardizing (mean 0 and variance 1) is important for PCA because it is looking for a new orthogonal basis where the origin ...


4

PCA simply finds more compact ways of representing correlated data. PCA does not explicitly compact the data in order to better explain the target variable. In some cases, most of your inputs might be correlated with each other but have minimal relevance to your target variable. That's probably what is happening in your case. Consider a toy example. Lets ...


4

PCA will not change the order of your points. The first point will still be the first point. As for the second, this is too unclear to answer. There is no obvious relationship between the number of clusters and the number of PCs. If you use too few PCs, your data approximation is too crude. If you use too many, then you will be working with random ...


4

Without putting in the time to look through Azure's documentation, my guess is that their PCA method is really just a way to do a feature reduction, then use some algorithm they have to classify. Best thing to do is try both methods and then CV and compare performances. https://gallery.azure.ai/Experiment/1219e87f8fb84e88a2e1b54256808bb3


4

Having highly correlated features is a type of redundancy in features. And yes, it effects a regression model if you are having highly correlated features. A very nice explanation is given here. PCA is a nice choice when it comes to dimensionality reduction.


4

STEP 1 When you do Regression on this dataset, what you obtain is: $\hspace{60mm}f(x_1,x_2,x_3,x_4) = ERI$ where your $x_1,x_2,x_3,x_4, ERI$ are O2 level, Cylinder Pressure, Fuel Flow,, Engine Temp, and Engine Running Indexrespecetively. In case of Multiple Regression, you will get 1 coefficient per feature (total 4 in coef_, {$w_1,w_2,w_3,w_4$}) and one ...


4

Someone correct me if I'm wrong, but the PCA process itself doesn't assume anything about the distribution of your data. The PCA algorithm is simple - find the direction of greatest variance in your data write down the direction of the vector pointing in that direction, and 'divide' the data along that direction by its variance in that direction, so the ...


4

Neural networks are actually extremely effective at performing dimensionality reduction. A great example is word2vec, which applies a shallow neural network to reduce inputs on the order of several million features (i.e. unnormalized text) to a 30-150 dimensional vector via a process that is mathematically analogous to matrix factorization (which is the ...


4

PCA is used to abandon having redundant features. It expands directions which your data is highly distributed in those directions. During this process, it does not care about the labels of your data. In those directions your data will be highly distributed and none of features have correlation. PCA just does the preceding things. In that feature space your ...


Only top voted, non community-wiki answers of a minimum length are eligible