0
$\begingroup$

Background

I'm going to be releasing a package for R that does calculations involving extremely large sparse matrices (1,000,000 x 1,000,000 is the minimum for what we consider useful). For this initial release, it uses the Matrix package because it was simple to get up and running and has acceptable memory and computation performance for our minimum goals. I'm aware that R by default uses reference BLAS/LAPACK libraries that don't have any special optimizations enabled (SIMD, parallel computing). As far as I can tell, the Matrix package doesn't seem to be taking advantage of any other libraries that would be providing these optimizations either. For a future release of my package, I would like to change this, because it would make it drastically more useful for end-users.

Limitations

The problem is that a large portion of the target audience for my package tends to lie at the extreme non-technical end of the spectrum (as in, getting R installed and maintained can sometimes be a significant challenge). Asking them to install alternatives like OpenBLAS or MKL is a no-go. Besides, for users that want to use the package on a remote installation, such as a supercomputer, this may not be an option anyway. Similar thing with MRAN; it might be easy to install as a way to get Microsoft's nice optimizations, but it will certainly cause its own headaches. No, ideally whatever solution I use, it should be mostly or entirely hidden from my users (basically, whatever is needed can be automatically installed with the package).

Potential Options

Rcpp is an option; I'm comfortable with C++, and already have some optimizations in mind that I could code in C++ to avoid a bunch of excess memory allocations. Enough prominent packages use it that I'm less concerned about it from an end-user perspective. So one option I'm looking at is RcppEigen. It appears that the Eigen library uses SIMD instructions, which is probably the biggest thing, but takes almost no advantage of parallel computing. I'm not as clear yet about the advantages/disadvantages with RcppArmadillo for my use case.

The Question

Using the Matrix package as my current standard, what are the best options for high performance (both in terms of speed and minimizing memory requirements) sparse matrix calculations in R that minimizes involving extra work from end-users?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.