Background:
This means a dataset of 7,000 samples and 3.2B columns, which I would have to read into distributed Spark memory somehow.
Obviously I want to reduce the number of columns that gets fed into the final model, although I could start with a smaller amount of samples.
Approaches:
I keep coming back to this Multiple Factor Analysis
for categorical features http://factominer.free.fr/factomethods/multiple-factor-analysis.html which looks like it would give me groups... but do I have to manually decide which groups to remove?
I've also read about autoencoders
NNs being used for dimensionality reduction.
I've also read about including all inputs directly in the first layer of my desired model and then drastically reducing the number of inputs into the second layer as another form of dimensionality reduction.
More Background:
There are 3.2B positions in the human genome. Each position can contain either one of 4 different nucleotides or some larger insertions/ deletions of nucleotides.
I have 2,200 cases (prior to QC filtering) for a relatively common disease and I can and compare them to as many controls as I like so 4,800 controls seems like a reasonable balance. I've read that case-control ratios edging toward 90-10 are considered imbalanced.
What approach should I take?