11
$\begingroup$

I'm currently working on a logistic regression model for genomics. One of the input fields I want to include as a covariate is genes. There are around 24,000 known genes. There are many features with this level of variability in computational biology and hundreds of thousands of samples are needed.

  • If I LabelEncoder() those 24K genes
  • and then OneHotEncoder() them ...

Is 24,000 columns going to make my keras training times unreasonable for a 2.2 GHz quad-core i7 CPU?

If so, is there a different approach to encoding that I can take with this?

Should I somehow try to dedicate a layer of my model to this feature?

Does this mean I need 24K input nodes?

Improve this question
$\endgroup$
1
  • $\begingroup$ Why not learn representation using VAE ? I do think in gene sequence learning the representation(like is done in NLP) will make a lot of sense compared with just a simple PCA ... $\endgroup$
    – n1tk
    Sep 5, 2019 at 1:36

3 Answers 3

Reset to default
10
$\begingroup$

Yes, using one-hot encoding on 24k features requires 24k input nodes. However this should not be a problem for Keras (or any other deep learning library). Natural language processing often uses one-hot encoding on words with a vocabulary size in the same ballpark.

If you are using a "deep" model, one of your hidden layers should take care of reducing the dimensionality of your data. A separate pre-processing step is usually not needed.

The training time should not be unreasonable.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you for the sanity check. $\endgroup$
    – Kermit
    Sep 3, 2019 at 1:25
  • $\begingroup$ I noticed you mentioned a layer, not PCA in sklearn. Would you recommend autoencoders as a form of dimensionality reduction? $\endgroup$
    – Kermit
    Sep 3, 2019 at 1:56
  • 2
    $\begingroup$ You mentioned you use Keras, so you are using some kind of neural network right? If somewhere in your network, you have a layer with a smaller number of nodes than your input, the network is performing dimensionality reduction automatically. I assume you only have one (or a small number of) regression outputs. So one simple way is to have the input layer (d=24k), one or more intermediate layers (d=1k or something like that) and your output layer (d=1). $\endgroup$
    – C. Yduqoli
    Sep 3, 2019 at 2:31
  • $\begingroup$ While similar size layers are common in NLP, it generally is unreasonable to train modern NLP models on CPU, powerful GPUs really are table stakes in NLP for anything beyond small 'toy' models. On the other hand, it's not a big barrier, as both physical and cloud GPUs are quite easily available. $\endgroup$
    – Peteris
    Sep 3, 2019 at 19:36
6
$\begingroup$

Entity Embedding for Categorical Variables (original pager) would be a very suitable approach here. Read on here, or here. I have actually put pieces of codes from here and there and made an complete running implementation, see this git repo. This easily handles very high cardinal categorical variables using neural networks. I won't list pros and cons of OHE, you just Google it, but one of its main drawbacks esp. when having a very high cardinal categorical variable is it increasing drastically your feature space unnecessarily, which in my opinion isn't ideal. And more importantly OHE, to my knowledge, doesn't account for semantic relationship between categories if exists such a relation! However, Entity Embedding is a concept to Word Embedding in NLP, the weights that are being learned to encode the categories can potentially capture intra-category relations.

Improve this answer
$\endgroup$
3
$\begingroup$

Generally, that number of genes should be reduced to much smaller set of meaningful features. Then, the reduced feature set can be used in the model. For example, Principal Component Analysis (PCA) is one of the most common reduction techniques and has been used for gene expression data.

"Machine learning for integrating data in biology and medicine: Principles, practice, and opportunities" by Zitnika et al. covers a variety of feature engineering techniques for genes.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you makes sense. Reduce feature space to the number of relevant genes (and regulatory zones in wgs). $\endgroup$
    – Kermit
    Sep 3, 2019 at 1:26
  • $\begingroup$ Is it too much of a noob question to ask what kind of dimensionality reduction you would recommend?: PCA, manifold, clustering/density, some kind of neural net? $\endgroup$
    – Kermit
    Sep 3, 2019 at 1:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.