# Linear Regression performance in R

I have a problem with R's performance: here is my Script.

The problem is that I need to use it in a base with ~6m of clients (with one linear model per customer) and it's taking to long to process.

Can anyone help me to improve the performance of my script? I think that the problem is in the data.frame rbind function.

• Have you looked at biglm? Sep 20, 2018 at 13:36
• On line 26 you are using a "for" loop...In general you should avoid this in R as it slows things down. Try to vectorise your function instead. Sep 20, 2018 at 14:19
• this is probably also very inefficient: results <- bind_rows(results, row) . You could preallocate a results vector with i elements
– knb
Sep 21, 2018 at 21:35
• You want to fit 6million regression models, one for each client? That's an trivially parallel problem and the data science approach would be to rent a few thousand Amazon EC2 systems for a short time. Sep 26, 2018 at 16:33

There's a few problems in your script actually, and growing objects (whether using rbind() or bind_rows() ) seriously messes up your performance. So if you know how many rows you have, start by creating an object with that many rows.

Next, you rely on a data frame. If you just store the info in separate vectors and only combine them in the end, you remove again a ton of overhead. Take a look at following three mockup examples:

library(rbenchmark)
library(dplyr)
# The naive way
f1 <- function(n){
out <- data.frame(x = numeric(), y = numeric())
for(i in seq.int(n)){
tmp <- data.frame(x = rnorm(1), y=rnorm(1))
out <- bind_rows(out, tmp)
}
return(out)
}
# Preallocating memory
f2 <- function(n){
out <- data.frame(x = numeric(n), y = numeric(n))
for(i in seq.int(n)){
out$$x[i] <- rnorm(1) out$$y[i] <- rnorm(1)
}
return(out)
}
# Using vectors
f3 <- function(n){
outx <- numeric(n)
outy <- numeric(n)
for(i in seq.int(n)){
outx[i] <- rnorm(1)
outy[i] <- rnorm(1)
}
return(data.frame(x = outx, y = outy))
}


On my machine, I get the following timings:

benchmark(
f1(100),
f2(100),
f3(100),
columns = c("test","elapsed","relative","replications")
)
##     test elapsed relative replications
## 1 f1(100)    2.16    27.00          100
## 2 f2(100)    0.34     4.25          100
## 3 f3(100)    0.08     1.00          100