0
$\begingroup$

I am subsetting some data frames and am sure there is a better way.

Essentially, I have two data frames. The first is the actual data. The second has some meta data and importantly a flag on whether or not the row is in the subset I am interested in. All I would like to do is pull out the subset and write a file.

For instance, here is my subset data frame:

     head(temp[,c(1,4:8)])
     ensembl_gene_id   FGF Secreted  SigP Receptors    TF
1 ENSMUSG00000000001 FALSE    FALSE FALSE      TRUE FALSE
2 ENSMUSG00000000001 FALSE    FALSE FALSE      TRUE FALSE
3 ENSMUSG00000000001 FALSE    FALSE FALSE      TRUE FALSE
4 ENSMUSG00000000001 FALSE    FALSE FALSE      TRUE FALSE
5 ENSMUSG00000000001 FALSE    FALSE FALSE      TRUE FALSE
6 ENSMUSG00000000001 FALSE    FALSE FALSE      TRUE FALSE

Here is my actual data:

Expt1_C1_EXONS_dxd_res_wDesc[1:5,1:5]
                                   groupID featureID exonBaseMean dispersion         stat
ENSMUSG00000000001:E001 ENSMUSG00000000001      E001    624.80240 0.04271781  1.255734504
ENSMUSG00000000001:E002 ENSMUSG00000000001      E002     30.92281 0.02036015  0.514038911
ENSMUSG00000000001:E003 ENSMUSG00000000001      E003     41.61413 0.01546023 10.105615831
ENSMUSG00000000001:E004 ENSMUSG00000000001      E004    137.47209 0.03975305  0.281105120
ENSMUSG00000000001:E005 ENSMUSG00000000001      E005     85.97116 0.05753662  0.005482149`

What I was doing is:

write.table(Expt1_C1_EXONS_dxd_res_wDesc[temp$FGF,],
            "Expt1_C1_EXONS_dxd_res_wDesc_FGF.tab",col.names=T,row.names=F,quote=F)

This is taking 20+ min per subset just to subset and write the data. The data frame is 370,000 rows by 27 variables, so large but not huge. I have about 30 of these to do. Is there a more efficient way? Note, the groupID does NOT equal the first column in my subset data frame. In some instances the groupID contains a concatenated set of ensembl ids. So I have preprocessed to get the temp data frame to have what I want in the same row order.

Thanks, Bob

Improve this question
$\endgroup$
1
  • $\begingroup$ thanks. It turns out the issue was Expt1_C1_EXONS_dxd_res_wDesc[temp$FGF,] was not doing what I intended. I had thought it would interpret the T/F in that column in the data frame temp and use that as inclusion/exclusion criteria. Changing that to Expt1_C1_EXONS_dxd_res_wDesc[temp$FGF=="TRUE",] works and takes almost no time. Examining the results of the incorrect version, it produced a huge file with all NA's. So my question is now, why didn't it take my temp$FGF values and what was it actually doing in that case? Thanks $\endgroup$
    – user8706
    Commented Mar 18, 2015 at 12:01

2 Answers 2

Reset to default
1
$\begingroup$

Only reason for such a long subset time is that you're running out of memory and the OS starts to swap. AFAIK base data.frame will create a copy of the data in memory before writing it to file.

What you can do is use either dplyr or data.table packages as they manage some of these steps without copying the data.

Improve this answer
$\endgroup$
0
$\begingroup$

Without looking into it too deeply (i don't have access to R on this computer), it seems...

temp is a data frame with x rows, Expt1_C1_EXONS_dxd_res_wDesc is a data frame with y rows. temp$FGF is therefore a vector of length x. Since temp is a subset, x < y

When you write Expt1_C1_EXONS_dxd_res_wDesc[temp$FGF,] you might be getting errors (NA's) due to the mismatch in length. It may also be because you are not specifying which rows you want, the vector temp$FGF contains TRUE and FALSE. It doesn't contain row indices. temp$FGF==TRUE returns (or at least selects based on) indices.

(As i say, i don't have R in front of me to check.)

Also, as far as i remember, you need to specify whether you want TRUE or FALSE entries to be included. R does not automatically assume you want TRUE.

(Again, i don't have R in front of me to check.)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.