# CDF with a massive amount of data

I am trying to plot a CDF of 7 files I have. Each file looks like this:

1
2
1.5
2.3


and so on. The problem is about the sizes of the files, which are:

862M
1,8G
2,4G
18G
2,0G
1,8G


I have put up a simple R script which simply loads the files and plot them. When I generate fake files of smaller size, the script works fine. However, it is running since three days and it hasn't produced any output using the full files.

The script is this:

library(ggplot2)
data$g = "G1" data2 <- read.table('file2.csv') data2$g = "G2"
data4$g = "G3" data4 <- read.table('file4.csv') data4$g = "G4"
data5$g = "G5" data6 <- read.table('file56.csv') data6$g = "G6"
data7\$g = "G7"

dftotal = rbind(data,data2)
dftotal = rbind(dftotal,data3)
dftotal = rbind(dftotal,data4)
dftotal = rbind(dftotal,data5)
dftotal = rbind(dftotal,data6)
dftotal = rbind(dftotal,data7)

gp <- ggplot(data = dftotal, aes(x = V1), group = factor(g)) + stat_ecdf()
ggsave('cdf.eps',gp)


Does anyone knows a more efficient way to do this?

• How much RAM memory does your computer have? Jun 1 '16 at 8:42
• 16 GB. But I am more worried about the time it takes than the memory consumption, so that's why I'm asking whether faster solutions exist. Jun 1 '16 at 9:07
• Well, you are loading and then sorting 27 GB of text of numbers, I feel (not sure if this is true) that you are going to walk into problems with regards to memory regardless. However even if that's not the case, the ecdf has to sort all those numbers. Can you tell us something about the distribution of these numbers? Jun 1 '16 at 9:12
• Not much, they are in the [0,50] interval, mostly condensed below 35. I am able to plot them individually, but together it takes too much time. I am wondering whether I can preprocess each file separately, and then plot them together. Maybe computing the x-y values and plot them instead of computing the ecdf directly on all the values. Jun 1 '16 at 9:16
• This sequence of rbinds may be not efficient. I think if you manually write a procedure for counting numbers in prespecified intervals, it would take the same time as to read those files without loading anything in memory. Jun 2 '16 at 8:21