I have the following dataset with the dimensions:
Rows: 41174
Columns: 439316
The matrix is very sparse and on this, I want to perform Dimensionality Reduction. I am using Spark's computeSVD function to perform the dimensionality reduction.
However, I get an error saying that
Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: k = 41174 and/or n = 439314 are too large to compute an eigendecomposition
But I ran the same computeSVD on the following dataset and it ran perfectly fine.
Rows: 3502
Columns: 103301
In both the cases, I am passing the value of "k" to be the Minimum of Rows, Columns. I am not able to understand what I am doing wrong here. As per the error, the issue is with K. How to resolve the above error. Also, any ideas on how to determine the K?