1
$\begingroup$

Background:

This means a dataset of 7,000 samples and 3.2B columns, which I would have to read into distributed Spark memory somehow.

Obviously I want to reduce the number of columns that gets fed into the final model, although I could start with a smaller amount of samples.

Approaches:

I keep coming back to this Multiple Factor Analysis for categorical features http://factominer.free.fr/factomethods/multiple-factor-analysis.html which looks like it would give me groups... but do I have to manually decide which groups to remove?

I've also read about autoencoders NNs being used for dimensionality reduction.

I've also read about including all inputs directly in the first layer of my desired model and then drastically reducing the number of inputs into the second layer as another form of dimensionality reduction.

More Background:

There are 3.2B positions in the human genome. Each position can contain either one of 4 different nucleotides or some larger insertions/ deletions of nucleotides.

I have 2,200 cases (prior to QC filtering) for a relatively common disease and I can and compare them to as many controls as I like so 4,800 controls seems like a reasonable balance. I've read that case-control ratios edging toward 90-10 are considered imbalanced.

What approach should I take?

|improve this question|||||
$\endgroup$
  • $\begingroup$ can these massive vectors be split? I know nothing of genomes. Are there logical ways you can decompose each "observation"? $\endgroup$ – Leevo Sep 12 '19 at 13:20
  • $\begingroup$ Yes. The columns are the same for each sample. That is to say, 10.000 samples could easily be split out into 10 groups of 1.000 samples. $\endgroup$ – HashRocketSyntax Sep 12 '19 at 13:22
  • 1
    $\begingroup$ I think it would be worth studying the field-specific literature and/or ask this question on biology.stackexchange.com. You would probably have to do some more specific pre-processing instead of trying to feed the whole genome to a ML system. $\endgroup$ – Erwan Sep 12 '19 at 14:58
0
$\begingroup$

Based on this article, I have decided to run with an autoencoder (tanh, tanh, tanh) neural net as opposed to PCA.

"[for] thousands of dimensions the autoencoder has a better chance of unpacking the structure and storing it in the hidden nodes by finding hidden features ... [Also,] in contrast to PCA, the autoencoder has all the information from the original data compressed in to the reduced layer. [Finally,] the autoencoder is better at reconstructing the original data set than PCA when k is small, however the error converges as k increases. For very large data sets this difference will be larger and means a smaller data set could be used for the same error as PCA."

https://www.r-bloggers.com/pca-vs-autoencoders-for-dimensionality-reduction/

Autoencoders for dimensionality reduction are also explained in OReilly's TensorFlow book.

UPDATE: It looks like an embeddings layer as the first hidden layer would also be a good start to reducing dimensionality.

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.