0
$\begingroup$

I am building machine learning algorithms in my laptop. It has i3 procesor and 16 GB RAM. Despite using multiple cores(3 out of 4), it takes 2 days to run all the techniques that i am trying to run an obviously data is huge (close to 1.3 million rows and 20 variables).

Is there any way to reduce this time required for running algorithms into fraction of the time it takes currently? Lets say some hours instead of days? I have heard from a friend who has computer science background that spark takes less time than stand alone R. Not sure if Spark can reduce the analysis time from multiple days to some hours. I am open to suggestions and solutions(preferably open source). Thoughts?

I am sure a solution for this must exist as R is pretty old and some genius would have found a way to solve this painful problem.

Improve this question
$\endgroup$
5
  • $\begingroup$ There is something you may want to try - "sample" $\endgroup$
    – karthikbharadwaj
    Jun 28 '16 at 1:23
  • $\begingroup$ What if my scenario does not allow me to do so! $\endgroup$
    – StatguyUser
    Jun 28 '16 at 1:44
  • $\begingroup$ Despite having multiple cores, have you registered your cores for parallel processing?. If you haven't R still uses only a single core for your processing. $\endgroup$
    – karthikbharadwaj
    Jun 28 '16 at 3:50
  • $\begingroup$ If you are going to do more analysis, I would recommend you to use octave. It out performs R. BUT: There are some edge cases where R is better. You can give it a try ;) IMHO $\endgroup$
    – Erhard Dinhobl
    Jun 28 '16 at 10:34
  • $\begingroup$ @karthikbharadwaj below is my code. library("parallel") library("foreach") library("doParallel") ##leave one cluster for other things than R cl <- makeCluster(detectCores() - 1) registerDoParallel(cl, cores = detectCores() - 1) $\endgroup$
    – StatguyUser
    Jun 28 '16 at 12:27
1
$\begingroup$

Depends on the models you are trying to run. Your data isn't that big. For example using a support vector model from the kernlab package, you run into problems. Not every model is fast or has a fast implementation.

Without more information on what you are doing it is difficult to say what causes the bottleneck. But if you just want a speed boost in running models, have a look at the xgboost package, the h2o package (GLM, GBM, rf, deeplearning), ranger for a faster implementation of a randomforest model.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks! I am using support vector machine, neural network apart from traditional techniques such as linear regression and couple of other techniques for a regression problem. I work in a challenging environment where quick turn around is expected. So i was exploring how to best utilize existing resources and if anyone has found solution to this problem so far. So my question is can spark do things faster than R in the same machine if i use SparkR? Or anything else can be done to fasten things? $\endgroup$
    – StatguyUser
    Jun 27 '16 at 15:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.