Hello and thanks in advance! I'd like some advice on a scalability issue and the best way to resolve. I'm writing an algorithm in R to produce forecasts for several thousand entities. One entity takes about 43 seconds to generate a forecast and upload the data to my database. That equates to about 80+ hours for the entire set of entities and that's much too long.
I thought about running several R processes in parallel, possibly many on a few different servers, each performing forecasts for a portion of total entities. Though that would work, is there a better way? Can Hadoop help at all? I have little experience with Hadoop so don't really know if it can apply. Thanks again!