3
$\begingroup$

Reducing dimensionality via PCA before training is a common practice, but PCA cannot makes use of nonlinear relations between features.

I read about UMAP (e.g. https://adanayak.medium.com/dimensionality-reduction-using-uniform-manifold-approximation-and-projection-umap-4aa4cef43fed), a technique for reducing dimensionality that is able to make sense of nonlinear relations between features.

However, I only saw its use in data presentation and exploration. Would it make sense to use UMAP as a form of feature engineering/dimensionality reduction when creating input for downstream model training?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Yes, it makes sense, and that is one of the advantages UMAP has over t-SNE. While t-SNE has no ability to operate on out-of-sample data, UMAP creates a map to the lower-dimension space that can be applied to out-of-sample data just like the PCA matrix would be applied to out-of-sample data.

(Certainly we could run everything through the t-SNE algorithm and then do the data split, but that is majorly cheating. What happens when we get new observations that didn’t exist when we built the model, like how Siri is supposed to be able to understand speech by people who have yet to be born when they can talk in a few years?)

Improve this answer
$\endgroup$
3
  • $\begingroup$ If had to use t-SNE I'll had to re-run t-SNE, get radically different features, and retrain the model each time. In some cases this might be possible, but still not a thing I would do. In most user cases, like your Siri example - that would be impossible. $\endgroup$
    – Lafayette
    Commented Jun 6, 2021 at 12:47
  • $\begingroup$ Why would t-SNE unable to operate out-of-sample? $\endgroup$
    – SmallChess
    Commented Oct 1, 2022 at 19:25
  • $\begingroup$ @SmallChess t-SNE doesn’t generate a function; it just jiggles points around. Consequently, you have nothing to apply to new data the way that you could apply a linear transformation developed from PCA on the in-sample data. $\endgroup$
    – Dave
    Commented Oct 1, 2022 at 19:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.