For the most part I am told that I should be using these methods to 'discover structure in the data' and I understand what is supposed to be happening is that, in the high dimensional space, there is some lower dimensional manifold that the data occupies and projecting that into lower dimensional space is supposed to help me understand something about the data. But examples of "good" embeddings look like nonsense to me. Furthermore, there is usually a small (3-5) number of hyperparameters that I have little idea what to do with.
For a concrete example, here is a parameter walk-through for UMAP. In the linked section on the parameter
n_neighbors the author says
By the stage of n_neighbors=20 we have a fairly good overall view of the data showing how the various colors interelate to each other over the whole dataset.
When I look at that plot, I see different colors splattered all over the place. There are multiple groups of each color. At best, it appears that the more translucent points have collected in the upper right of the plot. I understand that this particular set of images is supposed to illustrate the effect of the
n_neighbors parameter on the plot. But so what?
For another example, there is a webpage describing how to effectively use t-SNE. There are a number of great examples in there. But, so far as I can tell, the overall take-away is that you can more-or-less adjust the t-SNE parameters until you get an embedding you like for some reason. Again, so what? To what end?
As far as I can tell, these manifold methods, act as kind of a Rorschach test or a data science version of a cloud shape game. To reiterate my titular question: How do I interpret low dimentional embeddings of high dimentional embeddings? What am I supposed to be seeing in these plots that I am evidently not seeing?