0
$\begingroup$

I have a medical dataset which looks like this:

patient_id disease_id 
1111111111  DISEASE:1
1111111111  DISEASE:2
1111111111  DISEASE:3
1111111111  DISEASE:4
1111111111  DISEASE:5 
1111111111  DISEASE:6
1111111111  DISEASE:6
1111111112  DISEASE:1
1111111112  DISEASE:2
1111111112  DISEASE:4
1111111113  DISEASE:1
1111111113  DISEASE:5

which I need to feed into a neural network/random forest model. So, the only natural data representation to feed into the models I thought of was:

    patient_id   DISEASE:1  DISEASE:2  DISEASE:3  DISEASE:4  DISEASE:5  DISEASE:6  ...
    11111111111     1           1           1           1           1        1     ...  
    11111111112     1           1           0           1           0        0     ...    
    11111111113     1           0           0           0           1        0     ...

But my dataset is very big (~50GB, 1.5 GB compressed) and has tons of disease_ids so that the reshaping this data in the most efficient way possible in R requires 11.7 TB of space in compressed in RDs format (I know this because I divided the dataset into 100 chunks and reshaping of a single one resulted in 117 GB heavy RDs file; merging 100 of these would produce something larger than 11.7TB).

Now, I have 5 dataset this big that I need to merge together, so I feel a bit stuck. I need to come up with a more efficient data representation but don't know how as I am dealing with categorical variables which will require 1-hot encoding. Can anyone suggest any alternative ways to deal with a data like this.

Improve this question
$\endgroup$
3
  • $\begingroup$ Probably hashing could be an option en.wikipedia.org/wiki/Feature_hashing. Combined with a data pipeline, treating data like yours should be no problem stanford.edu/~shervine/blog/…. $\endgroup$
    – Peter
    Apr 18 '20 at 11:51
  • 1
    $\begingroup$ As your patients probably don't each map to many diseases, you could look into sparse tensors. These usually only hold the shape, values, indices of those values, meaning all the zeros are left out, with the overhead of storing the index information. $\endgroup$
    – n1k31t4
    Apr 18 '20 at 12:38
  • $\begingroup$ This question cannot be answered with the information presented here. There is NO mention of how many features, observations, missing values, etc. NO mention of any exploratory data analysis! What is the research question? Could the question be limited to a very small # of features? Are you looking at outliers(rarities) or averages? Do any features have near zero-variance? Is there any correlation between features? What about LDA or PCA? The data cannot all be categorical? There is a big difference between 50e6 obs. & 1000 features Vs 1000 obs. & 50e6 feature. Sample your data for any ideas. $\endgroup$
    – mccurcio
    May 7 '20 at 3:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.