15
$\begingroup$

I understand from Hinton's paper that T-SNE does a good job in keeping local similarities and a decent job in preserving global structure (clusterization).

However I'm not clear if points appearing closer in a 2D t-sne visualization can be assumed as "more-similar" data-points. I'm using data with 25 features.

As an example, observing the image below, can I assume that blue datapoints are more similar to green ones, specifically to the biggest green-points cluster?. Or, asking differently, is it ok to assume that blue points are more similar to green one in the closest cluster, than to red ones in the other cluster? (disregarding green points in the red-ish cluster)

enter image description here

When observing other examples, such as the ones presented at sci-kit learn Manifold learning it seems right to assume this, but I'm not sure if is correct statistically speaking.

enter image description here

EDIT

I have calculated the distances from the original dataset manually (the mean pairwise euclidean distance) and the visualization actually represents a proportional spatial distance regarding the dataset. However, I would like to know if this is fairly acceptable to be expected from the original mathematical formulation of t-sne and not mere coincidence.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ The blue points are the closest to their respective neighbor green points, this is how the embedding was performed. Loosely speaking the similarities (or distance) should be preserved. Going from 25 dimensions to only 2 very likely results in loss of information, but the 2D representation is the closest that can be shown on the screen. $\endgroup$
    – Vladislavs Dovgalecs
    Commented Mar 21, 2016 at 23:50

1 Answer 1

Reset to default
5
$\begingroup$

I would present t-SNE as a smart probabilistic adaptation of the Locally-linear embedding. In both cases, we attempt to project points from a high dimensional space to a small one. This projection is done by optimizing the conservation of local distances (directly with LLE, preproducing a probabilistic distribution and optimizing the KL-divergence with t-SNE). Then if your question is, does it keep global distances, the answer is no. It will depend on the "shape" of your data (if the distribution is smooth, then distances should be somehow conserved).

t-SNE actually doesn't work well on the swiss roll (your "S" 3D image) and you can see that, in the 2D result, the very middle yellow points are generally closer to the red ones than the blue ones (they are perfectly centered in the 3D image).

An other good example of what t-SNE does is the clustering of handwritten digits. See the examples on this link:https://lvdmaaten.github.io/tsne/

Improve this answer
$\endgroup$
6
  • 2
    $\begingroup$ What I mean is that you can't just use distance in the lower space as a similarity criterion. t-SNE will keep the global structure such as clusters but doesn't necessary keeps distances. This will depend on the shape of the high dimensional data and the perplexity you use. $\endgroup$
    – Robin
    Commented Mar 22, 2016 at 12:07
  • 1
    $\begingroup$ Ok I see. Thanks for clarifying. Yes I agree that distances in lower space would not be accurate. Now, since t-sne is practical for visualization can I use distances in the lower dimensional plot conceptually? For example in my plot can I say with certainty that blue points are closer or more similar to green ones than to red ones, given the obvious separation of the three groups in the 2d space. Or that would be also hard to say? $\endgroup$
    – Javierfdr
    Commented Mar 22, 2016 at 13:09
  • 1
    $\begingroup$ It pretty hard to say. The points in the low dimensional space are initialized with a gaussian distribution centered on the origin. They are then iteratively replaced optimizing the KL-divergence. So I would say that in your case blue points are more similar to the green cluster but there is now way to evaluate how closer they are than to the red cluster. t-SNE. $\endgroup$
    – Robin
    Commented Mar 22, 2016 at 22:52
  • 1
    $\begingroup$ Taken together, t-SNE puts emphasis on (1) modeling dissimilar datapoints by means of large pair-wise distances, and (2) modeling similar datapoints by means of small pairwise distances. Specifically, t-SNE introduces long-range forces in the low-dimensional map that can pull back together two (clusters of) similar points that get separated early on in the optimization. $\endgroup$
    – Robin
    Commented Mar 22, 2016 at 22:54
  • 1
    $\begingroup$ Very nice explanation. Thank you very much for your effort. I think that you different comments put together a complete answer. $\endgroup$
    – Javierfdr
    Commented Mar 23, 2016 at 20:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.