1
$\begingroup$

I have a problem with R's performance: here is my Script.

The problem is that I need to use it in a base with ~6m of clients (with one linear model per customer) and it's taking to long to process.

Can anyone help me to improve the performance of my script? I think that the problem is in the data.frame rbind function.

$\endgroup$
4
  • $\begingroup$ Have you looked at biglm? $\endgroup$ Sep 20, 2018 at 13:36
  • $\begingroup$ On line 26 you are using a "for" loop...In general you should avoid this in R as it slows things down. Try to vectorise your function instead. $\endgroup$
    – bstrain
    Sep 20, 2018 at 14:19
  • $\begingroup$ this is probably also very inefficient: results <- bind_rows(results, row) . You could preallocate a results vector with i elements $\endgroup$
    – knb
    Sep 21, 2018 at 21:35
  • 1
    $\begingroup$ You want to fit 6million regression models, one for each client? That's an trivially parallel problem and the data science approach would be to rent a few thousand Amazon EC2 systems for a short time. $\endgroup$
    – Spacedman
    Sep 26, 2018 at 16:33

1 Answer 1

2
$\begingroup$

There's a few problems in your script actually, and growing objects (whether using rbind() or bind_rows() ) seriously messes up your performance. So if you know how many rows you have, start by creating an object with that many rows.

Next, you rely on a data frame. If you just store the info in separate vectors and only combine them in the end, you remove again a ton of overhead. Take a look at following three mockup examples:

library(rbenchmark)
library(dplyr)
# The naive way
f1 <- function(n){
  out <- data.frame(x = numeric(), y = numeric())
  for(i in seq.int(n)){
    tmp <- data.frame(x = rnorm(1), y=rnorm(1))
    out <- bind_rows(out, tmp)
  }
  return(out)
}
# Preallocating memory
f2 <- function(n){
  out <- data.frame(x = numeric(n), y = numeric(n))
  for(i in seq.int(n)){
    out$x[i] <- rnorm(1)
out$y[i] <- rnorm(1)
  }
  return(out)
}
# Using vectors
f3 <- function(n){
  outx <- numeric(n)
  outy <- numeric(n)
  for(i in seq.int(n)){
    outx[i] <- rnorm(1)
    outy[i] <- rnorm(1)
  }
  return(data.frame(x = outx, y = outy))
}

On my machine, I get the following timings:

benchmark(
  f1(100),
  f2(100),
  f3(100),
  columns = c("test","elapsed","relative","replications")
)
##     test elapsed relative replications
## 1 f1(100)    2.16    27.00          100
## 2 f2(100)    0.34     4.25          100
## 3 f3(100)    0.08     1.00          100

Just changing this can seriously cut your running time already.

$\endgroup$
1
  • $\begingroup$ Thanks for your answer!! I'm going to try this solution. $\endgroup$ Oct 22, 2018 at 14:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.