1
$\begingroup$

I'm trying to run some analysis with some big datasets (eg 400k rows vs. 400 columns) with R (e.g. using neural networks and recommendation systems). But, it's taking too long to process the data (with huge matrices, e.g. 400k rows vs. 400k columns). What are some free/cheap ways to improve R performance?

I'm accepting packages or web services suggestions (other options are welcome).

Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ This is quite a broad question. You may be memory-bound (dataset doesn't fit into memory, and thus swap is used extensively, making it deadly slow) or CPU-bound (memory is ok, but operations just take too long). In latter case you can try to use vectorization more widely or write extensions in C directly. You can also try to compile your functions with cmpfun() or parallelize code to use several cores. Finally, you can rent Amazon web server to run your experiments, which will cost much less than buying hardware yourself. Anyway, try to determine your bottleneck first. $\endgroup$
    – ffriend
    Jul 8, 2014 at 6:18

2 Answers 2

Reset to default
6
$\begingroup$

Although your question is not very specific so I'll try to give you some generic solutions. There are couple of things you can do here:

  • Check sparseMatrix from Matrix package as mentioned by @Sidhha
  • Try running your model in parallel using packages like snowfall, Parallel. Check this list of packages on Cran which can help you runnning your model in multicore parallel mode.
  • You can also try data.table package. It is quite phenomenal in speed.

Good reads:

  1. 11 Tips on How to Handle Big Data in R (and 1 Bad Pun)
  2. Why R is slow & how to improve its Performance?
Improve this answer
$\endgroup$
2
$\begingroup$

Since you mention you are building a recommendation system, I believe you have a sparse matrix which you are working on. Check sparseMatrix from Matrix package. This should be able to help you with storing your large size matrix in memory and train your model.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.