3
$\begingroup$

So, I have a dataset with 39.949 variables and 180 rows. dataset is successfully saved in DataFrame but when I try to find cov() it result an error. here is the code

  import pandas as pd
  cov_data=pd.DataFrame(dataset).cov()

Here is the error

File "/home/syahdeini/Desktop/FP/pca_2.py", line 44, in find_eagen
cov_data=pd.DataFrame(data_mat).cov()
File "/usr/lib/python2.7/dist-packages/pandas/core/frame.py", line 3716, in cov
baseCov = np.cov(mat.T)
File "/usr/lib/python2.7/dist-packages/numpy/lib/function_base.py", line 1766, in cov
return (dot(X, X.T.conj()) / fact).squeeze()
ValueError: array is too big.
$\endgroup$
  • $\begingroup$ Isn't this a programming question that should be on Stack OVerflow? Getting an error when computing a covariance is not data science. $\endgroup$ – Spacedman Jun 10 '14 at 8:15
6
$\begingroup$

Christopher is right about the size of the array. To be simplistic about it, if this translates to 1.6B floats, at 16 bytes per float (32-bit version; 64-bit is bigger), then you're trying to create an array of about 26 GB. Even if you have the RAM for that, I'd imagine that it's probably going to overload something else along the way.

(Maybe not, but generally speaking, any operations that are that computationally intensive should first raise the question of whether you are doing the right calculation in the first place. And if you do need to do something of that magnitude, you should then try to break it down into more manageable chunks that can be run in parallel or distributed across machines.)

But given that you are describing a very, very wide dataset (~40k columns x 180 rows), I wonder whether you really want to take the covariance matrix of the transposed dataset (so 180x180 = 32,400 covariances)? That would be a far more tractable problem, and it's easier to see how it might be useful.

In any case, you're probably far better off calculating each pairwise covariance (or at least, the vector of cov(x_i,x_k) for all x_k != x_i) at the point where you'll actually use it, rather than calculating a giant matrix initially then referring back to it later. Memory issues aside, it'll make your life much easier if you start running things in parallel, and will help ensure you don't waste resources on unnecessary calculations.

$\endgroup$
  • $\begingroup$ Thanks for your answer mate. unfortunately, if I transpose the matrix it will make the false calculation. $\endgroup$ – Aldy syahdeini May 30 '14 at 11:46
  • 1
    $\begingroup$ Can you explain why you have to calculate such a giant covariance matrix? Would be easier to find a workaround if you could shed light on the motivation. $\endgroup$ – Therriault Jun 3 '14 at 15:01
2
$\begingroup$

Since you have 39,949 variables, the covariance matrix would have about 1.6 billion elements (39,949 * 39,949 = 1,595,922,601). That is likely why you are getting that error.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.