18
$\begingroup$

Are there any meanings for the dimensions of a t-sne embedding? Like with PCA we have this sense of linearly transformed variance maximizations but for t-sne is there intuition besides just the space we define for mapping and minimization of the KL-distance?

$\endgroup$
  • 1
    $\begingroup$ Not sure if this really gets at what you are asking, but I believe the dimensions of t-sne really only depend on the separability of the data. The dimensions can change given the same dataset because this is a non-linear transformation. So the dimensions can only really be interpreted within the context of a given instance. Let me know if I'm wrong, it is an interesting question. $\endgroup$ – Hobbes Mar 2 '17 at 21:17
  • $\begingroup$ Maybe it is just boring old ℝ3 ? $\endgroup$ – Nitro Mar 5 '17 at 4:49
17
+50
$\begingroup$

The dimensions of the low dimensional space have no meaning. Note that the t-SNE loss function is solely based on the distances between points ($y_i$ and $y_j$) and probability distributions over those distances ($p_{ij}$ and $q_{ij}$):

$$ \frac{\delta C}{\delta y_i}=4 \sum_j(p_{ij} - q_{ij})(y_i-y_j)(1+||y_i -y_j||^2)^{-1} $$

Thus there is no projection from the whole high-dimensional space to the low-dimensional space, t-SNE only finds a mapping from a specific set of high-dimensional points to a specific set of low dimensional points. Because there is no function from one space to the other there is also no inherent meaning of the axes.

Things you can imagine to illustrate this:

  • Rotating or translating the high-dimensional or low-dimensional space does not influences distances between the points. Hence, t-SNE does not care about rotation or translation in both spaces. Thus there is not absolute interpretation of the axes.
  • The t-Student distribution has fat tails. This causes the low-dimensional representation to be invariant to changes in points that are far away in the high-dimensional space. This also causes that points that are far away in the high-dimensional space can be either reasonably far away, far away or really far away in the low dimensional space. In this sense it stretches certain parts of the low-dimensional axes (in any arbitrary direction).

That being said, t-SNE is primarily a visualization technique and its dimension reduction effectiveness for other purpose is not obvious (probably not suitable for clustering, feature extraction or feature selection).

Also: the paper.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.