# How to reduce dimensionality of 3.2B categorical features?

Background:

This means a dataset of 7,000 samples and 3.2B columns, which I would have to read into distributed Spark memory somehow.

Obviously I want to reduce the number of columns that gets fed into the final model, although I could start with a smaller amount of samples.

Approaches:

I keep coming back to this Multiple Factor Analysis for categorical features http://factominer.free.fr/factomethods/multiple-factor-analysis.html which looks like it would give me groups... but do I have to manually decide which groups to remove?

I've also read about autoencoders NNs being used for dimensionality reduction.

I've also read about including all inputs directly in the first layer of my desired model and then drastically reducing the number of inputs into the second layer as another form of dimensionality reduction.

More Background:

There are 3.2B positions in the human genome. Each position can contain either one of 4 different nucleotides or some larger insertions/ deletions of nucleotides.

I have 2,200 cases (prior to QC filtering) for a relatively common disease and I can and compare them to as many controls as I like so 4,800 controls seems like a reasonable balance. I've read that case-control ratios edging toward 90-10 are considered imbalanced.

What approach should I take?

• can these massive vectors be split? I know nothing of genomes. Are there logical ways you can decompose each "observation"? – Leevo Sep 12 '19 at 13:20
• Yes. The columns are the same for each sample. That is to say, 10.000 samples could easily be split out into 10 groups of 1.000 samples. – Kalanos Sep 12 '19 at 13:22
• I think it would be worth studying the field-specific literature and/or ask this question on biology.stackexchange.com. You would probably have to do some more specific pre-processing instead of trying to feed the whole genome to a ML system. – Erwan Sep 12 '19 at 14:58
• Perhaps even better than the bio Stack is bioinformatics: bioinformatics.stackexchange.com. – Dave Jun 10 '20 at 10:34

Based on this article, I have decided to run with an autoencoder (tanh, tanh, tanh) neural net as opposed to PCA.