Issue with Spark SVD

Question

I have the following dataset with the dimensions:

Rows: 41174

Columns: 439316

The matrix is very sparse and on this, I want to perform Dimensionality Reduction. I am using Spark's computeSVD function to perform the dimensionality reduction.

However, I get an error saying that

Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: k = 41174 and/or n = 439314 are too large to compute an eigendecomposition

But I ran the same computeSVD on the following dataset and it ran perfectly fine.

Rows: 3502

Columns: 103301

In both the cases, I am passing the value of "k" to be the Minimum of Rows, Columns. I am not able to understand what I am doing wrong here. As per the error, the issue is with K. How to resolve the above error. Also, any ideas on how to determine the K?

Answer 1

After reading the source code, I find out that the problem is nothing related to your memory, but the limit of integer.

In the source code, it shows both of $n\min(2k,n)$ and $\min(2k,n)*(\min(2k,n)+8)$ should be less than Integer.MAX_VALUE, which is $2^{31}-1=2147483647$. In your case, $n\min(2k,n)=36176793968>2147483647$, and $\min(2k,n)*(\min(2k,n)+8)=6781851888>2147483647$.

I'm no spark expert nor java export so maybe I can't give you a solution. But personally speaking your dataset is truly big!

References

https://docs.oracle.com/javase/7/docs/api/java/lang/Integer.html