I have a medical dataset which looks like this:
patient_id disease_id
1111111111 DISEASE:1
1111111111 DISEASE:2
1111111111 DISEASE:3
1111111111 DISEASE:4
1111111111 DISEASE:5
1111111111 DISEASE:6
1111111111 DISEASE:6
1111111112 DISEASE:1
1111111112 DISEASE:2
1111111112 DISEASE:4
1111111113 DISEASE:1
1111111113 DISEASE:5
which I need to feed into a neural network/random forest model. So, the only natural data representation to feed into the models I thought of was:
patient_id DISEASE:1 DISEASE:2 DISEASE:3 DISEASE:4 DISEASE:5 DISEASE:6 ...
11111111111 1 1 1 1 1 1 ...
11111111112 1 1 0 1 0 0 ...
11111111113 1 0 0 0 1 0 ...
But my dataset is very big (~50GB, 1.5 GB compressed) and has tons of disease_id
s so that the reshaping this data in the most efficient way possible in R requires 11.7 TB of space in compressed in RDs format (I know this because I divided the dataset into 100 chunks and reshaping of a single one resulted in 117 GB heavy RDs file; merging 100 of these would produce something larger than 11.7TB).
Now, I have 5 dataset this big that I need to merge together, so I feel a bit stuck. I need to come up with a more efficient data representation but don't know how as I am dealing with categorical variables which will require 1-hot encoding. Can anyone suggest any alternative ways to deal with a data like this.