1
$\begingroup$

I am trying to plot a CDF of 7 files I have. Each file looks like this:

1
2
1.5
2.3

and so on. The problem is about the sizes of the files, which are:

862M
1,8G
2,4G
18G
2,0G
1,8G

I have put up a simple R script which simply loads the files and plot them. When I generate fake files of smaller size, the script works fine. However, it is running since three days and it hasn't produced any output using the full files.

The script is this:

library(ggplot2)    
data <- read.table('file1.csv')
data$g = "G1"
data2 <- read.table('file2.csv')
data2$g = "G2"
data3 <- read.table('file3.csv')
data4$g = "G3"
data4 <- read.table('file4.csv')
data4$g = "G4"
data5 <- read.table('file5.csv')
data5$g = "G5"
data6 <- read.table('file56.csv')
data6$g = "G6"
data7 <- read.table('file7.csv')
data7$g = "G7"

dftotal = rbind(data,data2)
dftotal = rbind(dftotal,data3)
dftotal = rbind(dftotal,data4)
dftotal = rbind(dftotal,data5)
dftotal = rbind(dftotal,data6)
dftotal = rbind(dftotal,data7)

gp <- ggplot(data = dftotal, aes(x = V1), group = factor(g)) + stat_ecdf()
ggsave('cdf.eps',gp)

Does anyone knows a more efficient way to do this?

Improve this question
$\endgroup$
5
  • $\begingroup$ How much RAM memory does your computer have? $\endgroup$
    – Jan van der Vegt
    Commented Jun 1, 2016 at 8:42
  • $\begingroup$ 16 GB. But I am more worried about the time it takes than the memory consumption, so that's why I'm asking whether faster solutions exist. $\endgroup$
    – lbedogni
    Commented Jun 1, 2016 at 9:07
  • $\begingroup$ Well, you are loading and then sorting 27 GB of text of numbers, I feel (not sure if this is true) that you are going to walk into problems with regards to memory regardless. However even if that's not the case, the ecdf has to sort all those numbers. Can you tell us something about the distribution of these numbers? $\endgroup$
    – Jan van der Vegt
    Commented Jun 1, 2016 at 9:12
  • $\begingroup$ Not much, they are in the [0,50] interval, mostly condensed below 35. I am able to plot them individually, but together it takes too much time. I am wondering whether I can preprocess each file separately, and then plot them together. Maybe computing the x-y values and plot them instead of computing the ecdf directly on all the values. $\endgroup$
    – lbedogni
    Commented Jun 1, 2016 at 9:16
  • $\begingroup$ This sequence of rbinds may be not efficient. I think if you manually write a procedure for counting numbers in prespecified intervals, it would take the same time as to read those files without loading anything in memory. $\endgroup$
    – Valentas
    Commented Jun 2, 2016 at 8:21

2 Answers 2

Reset to default
1
$\begingroup$

My estimate is that there are around 5.4b numbers in your dataset, all in the range of 0-50 according to your comment. I doubt there are much faster ways to do this on all the data than you are currently doing. However if you would just take every file and get random samples of 0.1-10% of each file and then combine these, you will still get an ecdf that is visually almost exactly the same while reducing memory issues and computing power necessary.

Improve this answer
$\endgroup$
0
0
$\begingroup$

If the frequency per bin is all you need, use awk or write a program in C to calculate that. The execution shouldn't take more than a few tens of minutes at most.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.