For simplicity I propose the following scheme:

  1. I have two DataFrames, one with genes, the other with miRNA (it is a simple example, it is not the case what the DF are), the data are continuous:
             Gene_1    Gene_2    Gene_3                                        
Patient_1    220.43    12,959    12,311
Patient_2    270.27    12,870    13,234
             miRNA_1   miRNA_2    miRNA_3                                        
Patient_1    220.43    12,959     12,311
Patient_2    270.27    12,870     13,234
  1. I must draw correlations all the genes against all the miRNAs (using either Pearson, Kendall or Spearman correlation, it doesn't matter) establishing as final result the following structure:
Gene     miRNA      Correlation  P-value        Adjusted P-value
Gen_1    miRNA_1    0,959        0.00311        0.00014
Gen_1    miRNA_2    -0,039       0.00311        0.00014
Gen_1    miRNA_3    -0,344       0.00311        0.00014
Gen_2    miRNA_1    0,1333       0.00311        0.00014
Gen_2    miRNA_2    0,877        0.00311        0.00014


The result of the Cross Join (all against all) can result in a DataFrame with billions of rows. To give a dimension of the required space, leaving aside the columns of the gene and the miRNAs and considering a small result of 300 million rows would need 300000000 * (16 bits of the correlation in floating pt + 64 bits of p-value + 64 bits of adjusted p-value) = 5 GB approx.

In order to optimize the memory usage, I do the computation in batches, the problem is with the adjusted p-values since I use the method of Benjamini & Hochberg (1995) with the function p.adjust of R (using a Python wrapper) that requires the complete p-value array, which makes me run out of memory.

Is there any way to compute, either from another library or another similar statistical method, the p-value adjusted in batches?

I already tried the FastLSU technique to be able to filter out some p-values that are not significant, but as I understood it, when I get the adjusted p-value for each row I need to know all the total p-values. If I could get for each row the adjusted p-value my problem would be solved since I could download the results of the batches to disk and goodbye to complications.

If someone could shed some light on the subject I would be very grateful


Can you explain further about how you are computing in batches? The entire p-value array itself should not be large enough to cause the memory problem. So it seems like your main problem is that you are trying to hold the cross-joined DF in memory and do the computation you need. Therefore these are my thoughts:

  • You might be able to solve this problem the way you are currently doing it if you have access to a cluster which has more memory.
  • There is another way to solve it that will use less memory and could work locally but will take a long time using nested loop (see below). You can also write results intermediately instead of doing a union at the end of each loop. Afterwards you can take the initial DFS out of memory, and then load in all of the intermediary results and create the final df.
  • You could use the method from prior bullet point in cluster which will be a bit better.
  • It would be even better if you could use a cluster and parallelize it. This problem would be well suited for Spark.
  • If you don't have access to a cluster, you can still parallelize it on your own computer which may help a little with speed. Again, I think if you use Spark and parallelize it on your computer it would be a bit faster.

Here I will explain the first bullet which I think will at least allow you to get your final data frame without the memory issue: I would first try making this problem into a nested loop of the columns of the data frames where you find the pvals for each gene --> all miRNAs and then subset this after computing q-values and using a cutoff. Then you will only be storing in memory the pairs which have met your q-value threshold. I would not suggest trying to create a joined DF with all of this data as that will take up too much memory. I wrote pseudo-code for a loop below. I feel like this would help with you running out of memory, however it will still be pretty slow. If you are still running out of memory, do you have a cluster available to you? Also it would be even better to parallelize this. You could write it in python but do you have Spark? If you will be continuing to work with 'big data' I would try to get access to a cluster at your institution or pay for it using AWS or something.

I am going to call your DF with the genes: geneDF and your DF with microRNA mirnaDF. Keep in mind this is python pseudo-code, and is pretty messy. I can clean it up if you think it would be useful to you. I more-so just wrote it to make what I was writing about clearer.

top_adjusted_pairs = pd.DataFrame(columns=['gene', 'miRNA', 'qval'])
for i in range(len(geneDF)):
    # initialize array for the distances between all microRNAs with just gene X
    miRNA_geneX_pvals = []*len(microRNAsDF)
    geneCol = geneDF[i]
    for j in range(len(mirnaDF)):
         mirnaCol = mirnaDF[j]
         # compute distances and write pvalue into array
         pval = dist(geneCol, mirnaCol).pval
         miRNA_geneX_pvals[j] = [miRNA.name, gene.name, pval]
    # now that you have the array of distances between gene X and all miRNAs you can use multiple hypothesis correction. 
    qvals = adjust(miRNA_geneX_pvals['pval'])
    # pick q-val cut off
    top_mirna_geneX = miRNA_geneX_pvals[qvals<0.1]

| improve this answer | |
  • $\begingroup$ Hello! First of all, thank you very much for responding. We actually do the computation in batches, the problem with the p-value adjustment is that the BH method requires the complete array (at least the p.adjust function of R that we are using). Do you know any way to do the adjustment only with the p-values you have in memory (the batch)? The example we give is small, it could be billions of p-values and it ends up throwing memory errors by having to adjust everything in one go $\endgroup$ – Genarito Jul 10 at 20:45
  • $\begingroup$ There is a question in Stack Overflow that touches on the subject, but does not provide any specific algorithm. And yes, we are considering using Spark, but it would be ideal to be able to develop a solution on a single machine as it would allow us to address the problem in the most optimal way in memory $\endgroup$ – Genarito Jul 10 at 20:47
  • $\begingroup$ Are you trying to do the adjustment on every possible miRNA,gene pair all together? I think that you could make an argument for doing the adjustments separately. so like if say you have n miRNA (miRNA_1, miRNA_2, ... ,miRNA_n) and m genes (gene_1, gene_2, ..., gene_m). Maybe it would be helpful if you could explain the problem more but I think you can do BH m times on m different arrays where one array would be the p-values for the following pairs: ( (gene_1, miRNA_1), (gene_1, miRNA_2), .... (gene_1, miRNA_m)). $\endgroup$ – fractalnature Jul 10 at 20:55
  • 1
    $\begingroup$ I think there is an argument to consider each gene separately, which means that you could do adjustment m times on m arrays where each of those arrays is the pairs between a gene and all of the other miRNAs. it wold be like saying, I have n hypotheses to test to see if gene_1 is correlated with any of the possible miRNAs. Find pvals, do BH on this array. Then repeat (a) for all genes. $\endgroup$ – fractalnature Jul 10 at 21:03
  • $\begingroup$ If the point of mulitple hypothesis correction is to deal with creation of false positives by creating too many tests, then I think you could argue that corr(gene_1, miRNA_1), corr(gene_100, miRNA50) should not be compared but corr(gene_100,miRNA50) can be compared with corr(gene_100, miRNA1050) $\endgroup$ – fractalnature Jul 10 at 21:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.