What can we learn from PCA on non linear data?

Question

Suppose we have dataset with 10 features which are not linear:

    import numpy as np
    from sklearn.decomposition import PCA
    import matplotlib
    import matplotlib.pyplot as plt
    
    v1 = np.random.rand(100)
    
    print (type(v1))
    
    v2 = 2**v1
    v3 = 3**v1 + np.matmul(v1, v1)
    v4 = 4**v1 + np.matmul(v2, v3)
    v5 = 5**v1 + np.matmul(v1, v3)
    v6 = 6**v1 + np.matmul(v1, v4)
    v7 = 7**v1 + np.matmul(v2, v2)
    v8 = 8**v1 + np.matmul(v4, v5)
    v9 = 9**v1
    v10 = 10**v1
    
    v = [v1,v2, v3, v4,v5, v6,v7, v8, v9,v10]
   
    pca = PCA()
    pca.fit(v)
    pca.explained_variance_ratio_
    
    PC_values = np.arange(pca.n_components_) + 1
    plt.plot(PC_values, pca.explained_variance_ratio_, 'ro-', linewidth=2)
    plt.title('Scree Plot')
    plt.xlabel('Principal Component')
    plt.ylabel('Proportion of Variance Explained')
    plt.show()

• I know that PCA is used to find the linear correlation. But what can we learn from that example?
• Can we use the PCA results and use only the first component of the PCA to train-predict our model?
• Does the result shown here is valid (correct for further processing)?

Answer 1

Not exactly an answer but just to say that, PCA is not a great way to approach non-linear relationships in your features. Imagine, trying to fit a linear regression to non linearly distributed data, like in the example below (only difference is PCA tries to minimise sum of squared distances conversely to residual sum of squares):

Nonetheless, there is the chance PCA will be able to help measuring some of the data variance however keep in mind that even if no correlations exist in your dataset PCA will return PCs ranked in order of variance in each of your features.

I would suggest using kernel PCA, especially if you know there are non-linear relationships in your features.

Answer 2

There is another technique that works with non linear relations. Is called T-distributed Stochastic Neighbor Embedding - tSNE(https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html)

The author (Laurens van de Maaten) wrote a blog some time ago that I found very useful https://lvdmaaten.github.io/tsne/

t-SNE is a tool to visualize high-dimensional data. It converts similarities between data points to joint probabilities and tries to minimize the Kullback-Leibler divergence between the joint probabilities of the low-dimensional embedding and the high-dimensional data. t-SNE has a cost function that is not convex, i.e. with different initializations we can get different results.

Answering your questions

• I know that PCA used to find linear correlation. But what can we learn from that example ?

PCA finds linear correlation. Many features have a monotonic transformation, so there is a linear relation. Those who are not, can not be reduced with PCA. That figure is telling you that all that PCA finds, is what is in the first feature. Which makes sense given how you have created the variables.

• Can we use the PCA results and use only the first component of the PCA to train-predict our model?

I would not advise on that. If your model captures nonlinear dependencies, it will loose performance. I would suggest you make the experiment so things are clearer for you.

• Does the result show here are valid (correct for further processing)?

Seems valid. You are applying PCA. You are getting PCA.