Linear Regression performance in R

Question

I have a problem with R's performance: here is my Script.

The problem is that I need to use it in a base with ~6m of clients (with one linear model per customer) and it's taking to long to process.

Can anyone help me to improve the performance of my script? I think that the problem is in the data.frame rbind function.

Answer 1

There's a few problems in your script actually, and growing objects (whether using rbind() or bind_rows() ) seriously messes up your performance. So if you know how many rows you have, start by creating an object with that many rows.

Next, you rely on a data frame. If you just store the info in separate vectors and only combine them in the end, you remove again a ton of overhead. Take a look at following three mockup examples:

library(rbenchmark)
library(dplyr)
# The naive way
f1 <- function(n){
  out <- data.frame(x = numeric(), y = numeric())
  for(i in seq.int(n)){
    tmp <- data.frame(x = rnorm(1), y=rnorm(1))
    out <- bind_rows(out, tmp)
  }
  return(out)
}
# Preallocating memory
f2 <- function(n){
  out <- data.frame(x = numeric(n), y = numeric(n))
  for(i in seq.int(n)){
    out$x[i] <- rnorm(1)
out$y[i] <- rnorm(1)
  }
  return(out)
}
# Using vectors
f3 <- function(n){
  outx <- numeric(n)
  outy <- numeric(n)
  for(i in seq.int(n)){
    outx[i] <- rnorm(1)
    outy[i] <- rnorm(1)
  }
  return(data.frame(x = outx, y = outy))
}

On my machine, I get the following timings:

benchmark(
  f1(100),
  f2(100),
  f3(100),
  columns = c("test","elapsed","relative","replications")
)
##     test elapsed relative replications
## 1 f1(100)    2.16    27.00          100
## 2 f2(100)    0.34     4.25          100
## 3 f3(100)    0.08     1.00          100

Just changing this can seriously cut your running time already.