1
$\begingroup$

I have the following dataset with the dimensions:

Rows: 41174

Columns: 439316

The matrix is very sparse and on this, I want to perform Dimensionality Reduction. I am using Spark's computeSVD function to perform the dimensionality reduction.

However, I get an error saying that

Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: k = 41174 and/or n = 439314 are too large to compute an eigendecomposition

But I ran the same computeSVD on the following dataset and it ran perfectly fine.

Rows: 3502

Columns: 103301

In both the cases, I am passing the value of "k" to be the Minimum of Rows, Columns. I am not able to understand what I am doing wrong here. As per the error, the issue is with K. How to resolve the above error. Also, any ideas on how to determine the K?

Improve this question
$\endgroup$
2
  • $\begingroup$ Dimensionality reduction is a computationally heavy job and it is saying that the matrix size is huge and it might be warning that your memory won't be sufficient to perform this job. $\endgroup$
    – Kiritee Gak
    Mar 15, 2017 at 5:59
  • $\begingroup$ True. The dataset size is around 17G and the also we are using spark. My assumption is that the distributed nature of spark will handle the issue related to memory. If it helps, I have a RAM of 56Gb. $\endgroup$
    – user1683894
    Mar 15, 2017 at 6:03

1 Answer 1

Reset to default
1
$\begingroup$

After reading the source code, I find out that the problem is nothing related to your memory, but the limit of integer.

In the source code, it shows both of $n\min(2k,n)$ and $\min(2k,n)*(\min(2k,n)+8)$ should be less than Integer.MAX_VALUE, which is $2^{31}-1=2147483647$. In your case, $n\min(2k,n)=36176793968>2147483647$, and $\min(2k,n)*(\min(2k,n)+8)=6781851888>2147483647$.

I'm no spark expert nor java export so maybe I can't give you a solution. But personally speaking your dataset is truly big!

References

https://docs.oracle.com/javase/7/docs/api/java/lang/Integer.html

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.