0
$\begingroup$

I am trying to understand what I am supposed to learn about a problem when using dimensionality reduction methods. In particular, I am referring to methods like t-SNE and UMAP.

For the most part I am told that I should be using these methods to 'discover structure in the data' and I understand what is supposed to be happening is that, in the high dimensional space, there is some lower dimensional manifold that the data occupies and projecting that into lower dimensional space is supposed to help me understand something about the data. But examples of "good" embeddings look like nonsense to me. Furthermore, there is usually a small (3-5) number of hyperparameters that I have little idea what to do with.

For a concrete example, here is a parameter walk-through for UMAP. In the linked section on the parameter n_neighbors the author says

By the stage of n_neighbors=20 we have a fairly good overall view of the data showing how the various colors interelate to each other over the whole dataset.

When I look at that plot, I see different colors splattered all over the place. There are multiple groups of each color. At best, it appears that the more translucent points have collected in the upper right of the plot. I understand that this particular set of images is supposed to illustrate the effect of the n_neighbors parameter on the plot. But so what?

For another example, there is a webpage describing how to effectively use t-SNE. There are a number of great examples in there. But, so far as I can tell, the overall take-away is that you can more-or-less adjust the t-SNE parameters until you get an embedding you like for some reason. Again, so what? To what end?

As far as I can tell, these manifold methods, act as kind of a Rorschach test or a data science version of a cloud shape game. To reiterate my titular question: How do I interpret low dimentional embeddings of high dimentional embeddings? What am I supposed to be seeing in these plots that I am evidently not seeing?

$\endgroup$

1 Answer 1

1
$\begingroup$

With this kind of low dimensional embeddings, you confirm visually some kind of relationship between the data points that you suspected by intuition.

For instance, the following graph taken from the article From bilingual to multilingual neural-based machine translation by incremental training shows the UMAP 2D plot of the representations computed by a neural network for sentences in different languages. The plot shows that the different languages form clusters that are distant from each other, which makes us think that the neural network is not able to properly learn the same representation space for all language.

enter image description here

Another typical example is with MNIST, the handwritten digit dataset. Here you can see plots of the bottleneck representation of an autoencoder and from PCA from this blog post where each point is the 2D representation of a digit image from the dataset and its color is its label (i.e. which number it is):

enter image description here

In the figure, we can observe that, while PCA hardly separates the different numbers, the autoencoder indeed separates them.

$\endgroup$
3
  • $\begingroup$ I appreciate your response, but I don't get it. For the language example, it is nearly certainly possible to choose a different set of hyperparameters such that you can make all the colored dots occupy the same region in the embedding space. In the Autoencoder vs PCA example, I don't think the AE has separated the groups at all, they are are shaped differently and have a point of convergence (where all the groups overlap). The PCA plot certainly looks "worse," if I was hoping for distinct clusters, but I was not, so I think it looks like a fish swimming to the left. $\endgroup$ Jun 21, 2023 at 14:46
  • $\begingroup$ A more direct response to your answer might be that using an extremely flexible model to "confirm visually some kind of relationship between the data points that you suspected by intuition" isn't of much value. Maybe there is some theorem that says that if you can separate two groups in low dimensional space, you can also do it in the higher-dimensional space? $\endgroup$ Jun 21, 2023 at 14:48
  • 1
    $\begingroup$ I agree that this kind of visualization is proof of nothing (especially with t-SNE, which, in my experience, has high variability in the resulting plots). I understand, though, that it's just a visually-pleasing way of adding weak evidence to a point/hypothesis and complementing other stronger pieces of evidence. $\endgroup$
    – noe
    Jun 21, 2023 at 15:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.