2
$\begingroup$

I am using a three node system:

  • Master node: $64$ GB RAM
  • 2 slave node: $32$ GB RAM

I am calculating the inverse of a square matrix using inbuilt Apache Spark function SVD. When I called SVD function of Apache spark then $64$ GB memory of master node got totally consumed and start to use swap memory which makes execution too slow and eventually lead to heap memory full.

If matrix size is $3000 \times 3000$ then no issue but when we consider size greater than $3000$ (like $3500$ or $5000$) then above-mentioned issue arises.

Note: Even my MATLAB is able to compute the inverse of $10K \times 10K$ matrix on 32 GB RAM but Apache Spark got Memory full if I used inbuilt SVD function on above system.

Sample Code is as below:-

import org.apache.spark.mllib.linalg.{Vectors,Vector,Matrix,SingularValueDecomposition,DenseMatrix,DenseVector}
import org.apache.spark.mllib.linalg.distributed.RowMatrix

def computeInverse(X: RowMatrix): DenseMatrix = {
  val nCoef = X.numCols.toInt
  val svd = X.computeSVD(nCoef, computeU = true)
  if (svd.s.size < nCoef) {
    sys.error(s"RowMatrix.computeInverse called on singular matrix.")
  }

  // Create the inv diagonal matrix from S 
  val invS = DenseMatrix.diag(new DenseVector(svd.s.toArray.map(x => math.pow(x,-1))))

  // U cannot be a RowMatrix
  val U = new DenseMatrix(svd.U.numRows().toInt,svd.U.numCols().toInt,svd.U.rows.collect.flatMap(x => x.toArray))

  // If you could make V distributed, then this may be better. However its alreadly local...so maybe this is fine.
  val V = svd.V
  // inv(X) = V*inv(S)*transpose(U)  --- the U is already transposed.
  (V.multiply(invS)).multiply(U)
}
$\endgroup$
  • 1
    $\begingroup$ Please post your code (the most simplified version possible that still exposes the problem). $\endgroup$ – Pete Nov 16 '17 at 2:25
  • $\begingroup$ @Pete Sample code is included in the question. $\endgroup$ – Chandan Gautam Nov 18 '17 at 8:33
  • $\begingroup$ I wonder what matrix inversion algorithm Spark uses. The exact solution uses O(n^2) memory, which might be the cause of the problem. (arxiv.org/pdf/1505.07570.pdf) Because of this people sometimes use approximate inverses. $\endgroup$ – tom Nov 19 '17 at 6:49
  • $\begingroup$ Did you actually allocate most of the machine's memory to the driver? the default heap size isn't nearly that big. $\endgroup$ – Sean Owen Feb 23 at 1:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.