2
$\begingroup$

I'm working on the PCA of the mnist dataset, and I get a very strange result, I created a matrix whose rows are flattened mnist images, When I try to compute the eigenvalues of the covariance matrix, I get some negative values. But the covariance matrix is positive semi-definite.

np.linalg.eigvals(np.dot(mnistBis[:, 0:20].T, mnistBis[:, 0:20])) # mnistBis.shape=(60000, 784)

array([ 4.79599869e+02, -1.19628465e+02,  9.68398702e+01,  1.88726171e-01,
        0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,
        0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,
        0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00,
        0.00000000e+00,  0.00000000e+00,  0.00000000e+00,  0.00000000e+00])

Modification: Here is the full code

import tensorflow.keras.datasets.mnist as mnist
import matplotlib.pyplot as plt
import numpy as np

(x_train, y_train), (x_test, y_test) = mnist.load_data()
mnistBis = np.reshape(x_train, (-1, 28*28))
np.linalg.eigvalsh(np.dot(mnistBis[:, 0:20].T, mnistBis[:, 0:20]))

```
Improve this question
$\endgroup$
5
  • $\begingroup$ What is your input looking like? Do you have any missing values? stats.stackexchange.com/questions/315297/… $\endgroup$
    – Soerendip
    Commented Mar 26, 2021 at 19:26
  • $\begingroup$ @NikosM. what do you mean? this is an array of eigenvalues. $\endgroup$
    – maths soso
    Commented Mar 26, 2021 at 19:38
  • $\begingroup$ yeap, misread it $\endgroup$
    – Nikos M.
    Commented Mar 26, 2021 at 19:45
  • $\begingroup$ @Sören what do you mean by missing values? np.dot(x.T, x) is a covariance matrix, I'd love to show the data, but I don't know how to show you a 60000x20 matrix $\endgroup$
    – maths soso
    Commented Mar 26, 2021 at 19:49
  • $\begingroup$ missing values are gaps in your dataset and may be the origin of the negative eigenvalues. $\endgroup$
    – Soerendip
    Commented Mar 27, 2021 at 21:42

2 Answers 2

Reset to default
1
$\begingroup$

That is probably a result of a floating point error.

The matrix is 60,000 x 20 and sparse (mostly zeros). The result of the calculations are values very close to zero that are not correctly represented by the computer.

Improve this answer
$\endgroup$
0
$\begingroup$

You seem to be using a generic eignvalue solver. Perhaps your matrix is badly conditioned and the aglorithm cannot pick it up. Lots of zero eigenvalues with few massive ones certainly seem suspicious. Have you tried solvers specific for hermitian/symmetric matricies? eigh & eigvalsh

Also, I would z-score all your features, and drop all features with zero variance. This will set an upper bound on maximum possible eigenvalue since trace of your matrix will be the number of the features, and since positive definite matrix has only positive eigenvalues, the largest eigenvalue will be no larger than the trace.

Improve this answer
$\endgroup$
4
  • $\begingroup$ eigvalsh gives me the same eigenvalues, in reverse order. $\endgroup$
    – maths soso
    Commented Mar 27, 2021 at 6:58
  • $\begingroup$ makes me think something wrong with your matrix. if you subtract transpose from your matrix, do you get all zeros? is matrix real-valued? if you multiply the eigenvector with negative eigenvalue, do you get the negative eigenvalue? Small negative eigenvalues are ok - numerical errors. Large ones are strange $\endgroup$
    – Cryo
    Commented Mar 27, 2021 at 13:14
  • $\begingroup$ Better still,what are the eigenvalues of the symmetrized matrix? i.e. $\Re\left(A+A^T\right)/2$, where $A$ is the original matrix $\endgroup$
    – Cryo
    Commented Mar 27, 2021 at 13:45
  • $\begingroup$ I made a modification, see the full code to reproduce it. $\endgroup$
    – maths soso
    Commented Mar 27, 2021 at 14:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.