# Computing adjusted p-values in batches

## Data

For simplicity I propose the following scheme:

1. I have two DataFrames, one with genes, the other with miRNA (it is a simple example, it is not the case what the DF are), the data are continuous:
             Gene_1    Gene_2    Gene_3
Patient_1    220.43    12,959    12,311
Patient_2    270.27    12,870    13,234

             miRNA_1   miRNA_2    miRNA_3
Patient_1    220.43    12,959     12,311
Patient_2    270.27    12,870     13,234

1. I must draw correlations all the genes against all the miRNAs (using either Pearson, Kendall or Spearman correlation, it doesn't matter) establishing as final result the following structure:
Gene     miRNA      Correlation  P-value        Adjusted P-value
Gen_1    miRNA_1    0,959        0.00311        0.00014
Gen_1    miRNA_2    -0,039       0.00311        0.00014
Gen_1    miRNA_3    -0,344       0.00311        0.00014
Gen_2    miRNA_1    0,1333       0.00311        0.00014
Gen_2    miRNA_2    0,877        0.00311        0.00014
...


## Problem

The result of the Cross Join (all against all) can result in a DataFrame with billions of rows. To give a dimension of the required space, leaving aside the columns of the gene and the miRNAs and considering a small result of 300 million rows would need 300000000 * (16 bits of the correlation in floating pt + 64 bits of p-value + 64 bits of adjusted p-value) = 5 GB approx.

In order to optimize the memory usage, I do the computation in batches, the problem is with the adjusted p-values since I use the method of Benjamini & Hochberg (1995) with the function p.adjust of R (using a Python wrapper) that requires the complete p-value array, which makes me run out of memory.

Is there any way to compute, either from another library or another similar statistical method, the p-value adjusted in batches?

I already tried the FastLSU technique to be able to filter out some p-values that are not significant, but as I understood it, when I get the adjusted p-value for each row I need to know all the total p-values. If I could get for each row the adjusted p-value my problem would be solved since I could download the results of the batches to disk and goodbye to complications.

If someone could shed some light on the subject I would be very grateful

Can you explain further about how you are computing in batches? The entire p-value array itself should not be large enough to cause the memory problem. So it seems like your main problem is that you are trying to hold the cross-joined DF in memory and do the computation you need. Therefore these are my thoughts:

• You might be able to solve this problem the way you are currently doing it if you have access to a cluster which has more memory.
• There is another way to solve it that will use less memory and could work locally but will take a long time using nested loop (see below). You can also write results intermediately instead of doing a union at the end of each loop. Afterwards you can take the initial DFS out of memory, and then load in all of the intermediary results and create the final df.
• You could use the method from prior bullet point in cluster which will be a bit better.
• It would be even better if you could use a cluster and parallelize it. This problem would be well suited for Spark.
• If you don't have access to a cluster, you can still parallelize it on your own computer which may help a little with speed. Again, I think if you use Spark and parallelize it on your computer it would be a bit faster.

Here I will explain the first bullet which I think will at least allow you to get your final data frame without the memory issue: I would first try making this problem into a nested loop of the columns of the data frames where you find the pvals for each gene --> all miRNAs and then subset this after computing q-values and using a cutoff. Then you will only be storing in memory the pairs which have met your q-value threshold. I would not suggest trying to create a joined DF with all of this data as that will take up too much memory. I wrote pseudo-code for a loop below. I feel like this would help with you running out of memory, however it will still be pretty slow. If you are still running out of memory, do you have a cluster available to you? Also it would be even better to parallelize this. You could write it in python but do you have Spark? If you will be continuing to work with 'big data' I would try to get access to a cluster at your institution or pay for it using AWS or something.

I am going to call your DF with the genes: geneDF and your DF with microRNA mirnaDF. Keep in mind this is python pseudo-code, and is pretty messy. I can clean it up if you think it would be useful to you. I more-so just wrote it to make what I was writing about clearer.

top_adjusted_pairs = pd.DataFrame(columns=['gene', 'miRNA', 'qval'])
for i in range(len(geneDF)):
# initialize array for the distances between all microRNAs with just gene X
miRNA_geneX_pvals = []*len(microRNAsDF)
geneCol = geneDF[i]
# compute distances and write pvalue into array
pval = dist(geneCol, mirnaCol).pval
miRNA_geneX_pvals[j] = [miRNA.name, gene.name, pval]
# now that you have the array of distances between gene X and all miRNAs you can use multiple hypothesis correction.
$$$$
`