For simplicity I propose the following scheme:
- I have two DataFrames, one with genes, the other with miRNA (it is a simple example, it is not the case what the DF are), the data are continuous:
Gene_1 Gene_2 Gene_3 Patient_1 220.43 12,959 12,311 Patient_2 270.27 12,870 13,234
miRNA_1 miRNA_2 miRNA_3 Patient_1 220.43 12,959 12,311 Patient_2 270.27 12,870 13,234
- I must draw correlations all the genes against all the miRNAs (using either Pearson, Kendall or Spearman correlation, it doesn't matter) establishing as final result the following structure:
Gene miRNA Correlation P-value Adjusted P-value Gen_1 miRNA_1 0,959 0.00311 0.00014 Gen_1 miRNA_2 -0,039 0.00311 0.00014 Gen_1 miRNA_3 -0,344 0.00311 0.00014 Gen_2 miRNA_1 0,1333 0.00311 0.00014 Gen_2 miRNA_2 0,877 0.00311 0.00014 ...
The result of the Cross Join (all against all) can result in a DataFrame with billions of rows. To give a dimension of the required space, leaving aside the columns of the gene and the miRNAs and considering a small result of 300 million rows would need 300000000 * (16 bits of the correlation in floating pt + 64 bits of p-value + 64 bits of adjusted p-value) = 5 GB approx.
In order to optimize the memory usage, I do the computation in batches, the problem is with the adjusted p-values since I use the method of Benjamini & Hochberg (1995) with the function p.adjust of R (using a Python wrapper) that requires the complete p-value array, which makes me run out of memory.
Is there any way to compute, either from another library or another similar statistical method, the p-value adjusted in batches?
I already tried the FastLSU technique to be able to filter out some p-values that are not significant, but as I understood it, when I get the adjusted p-value for each row I need to know all the total p-values. If I could get for each row the adjusted p-value my problem would be solved since I could download the results of the batches to disk and goodbye to complications.
If someone could shed some light on the subject I would be very grateful