I have the following dataset with the dimensions:

Rows: 41174

Columns: 439316

The matrix is very sparse and on this, I want to perform Dimensionality Reduction. I am using Spark's computeSVD function to perform the dimensionality reduction.

However, I get an error saying that

Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: k = 41174 and/or n = 439314 are too large to compute an eigendecomposition

But I ran the same computeSVD on the following dataset and it ran perfectly fine.

Rows: 3502

Columns: 103301

In both the cases, I am passing the value of "k" to be the Minimum of Rows, Columns. I am not able to understand what I am doing wrong here. As per the error, the issue is with K. How to resolve the above error. Also, any ideas on how to determine the K?

  • $\begingroup$ Dimensionality reduction is a computationally heavy job and it is saying that the matrix size is huge and it might be warning that your memory won't be sufficient to perform this job. $\endgroup$ Mar 15 '17 at 5:59
  • $\begingroup$ True. The dataset size is around 17G and the also we are using spark. My assumption is that the distributed nature of spark will handle the issue related to memory. If it helps, I have a RAM of 56Gb. $\endgroup$ Mar 15 '17 at 6:03

After reading the source code, I find out that the problem is nothing related to your memory, but the limit of integer.

In the source code, it shows both of $n\min(2k,n)$ and $\min(2k,n)*(\min(2k,n)+8)$ should be less than Integer.MAX_VALUE, which is $2^{31}-1=2147483647$. In your case, $n\min(2k,n)=36176793968>2147483647$, and $\min(2k,n)*(\min(2k,n)+8)=6781851888>2147483647$.

I'm no spark expert nor java export so maybe I can't give you a solution. But personally speaking your dataset is truly big!




Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.