0
$\begingroup$

Hello and thanks in advance! I'd like some advice on a scalability issue and the best way to resolve. I'm writing an algorithm in R to produce forecasts for several thousand entities. One entity takes about 43 seconds to generate a forecast and upload the data to my database. That equates to about 80+ hours for the entire set of entities and that's much too long.

I thought about running several R processes in parallel, possibly many on a few different servers, each performing forecasts for a portion of total entities. Though that would work, is there a better way? Can Hadoop help at all? I have little experience with Hadoop so don't really know if it can apply. Thanks again!

$\endgroup$
1
  • $\begingroup$ I would begin by looking at the time required to make forecasts and interacting with server times. Can they be optimized further? If not, then you should try distributed/parallel computing solutions. $\endgroup$
    – Sidhha
    Feb 22, 2015 at 15:29

2 Answers 2

2
$\begingroup$

If you're working with R language, I would suggest first to try use R ecosystem's abilities to parallelize the processing, if possible. For example, take a look at packages, mentioned in this CRAN Task View.

Alternatively, if you're not comfortable or satisfied with the approaches, implemented by the above-referred packages, you can try some other approaches, such as Hadoop or something else. I think that a Hadoop solution would be an overkill for such problem, considering the learning curve, associated with it, as well as the fact that, as far as I understand, Hadoop or other MapReduce frameworks/architectures target long-running processes (an average task is ~ 2 hours, I read somewhere recently). Hope this helps.

$\endgroup$
0
$\begingroup$

If you know your approach is working, you can try to implement it more efficiently. Identify the crucial points, try to vectorize them better, for example.

The R interpreter isn't the fastest. There are some efforts underway, but they are not yet ready. Vectorization (which means less interpreter, more low-level code) often yields a factor of 2x-5x, but you can sometimes get a factor of 100x by implementing it e.g. in C. (And the R advocates are going to hate me for saying this...) Once you know that your approach is working, this may be worth the effort.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.