What does the long curve-shape t-SNE mean?

Question

I use 1-D CNN input 1*512 size time series data which randomly fragment segment, the output will classify input into 10 classes. After training the CNN, I apply t-SNE to the prediction which I fed in testing data. In general, the output shape of the tsne result is spherical(for example,applied on MNIST dataset). But now I apply t-SNE on my own dataset. No matter how I adjust perplexity early, learning rate or maximum number of iterations. It will give me the result of long-shaped output, just like the pic below. Does the long-shaped t-SNE mean anything? Thanks everyone beforehand.

Edit: Explaining about dataset contents. p.s. before feeding into CNN for training, I randomly split it into training/testing dataset.

Answer 1

It would make sense that the time-series data sticks together - and so forms these lines you are seeing. In normal time-series analysis where the variables are assumed to be random (e.g. modelled on Brownian motion), the best prediction for tomorrow is just the same as today. t-SNE finds the closest points withing your feature-space and embedding them into a 2D space. It is quite impressive that it picks it out and ends up with your plot!

While you often get circular looking plots, it is not true that you always get circles/spheres. t-SNE is maximising the distance between clusters and at the same to minimising the distance between points within a single cluster... for the sake of efficiency, circles arise very often. You can observe this in nature: the shape of planets, of bubbles in water... circles are efficient!

t-SNE does not allow you to directly interpret the distance between clusters back to the input units (e.g. a line twice as high as another doesn't mean the values are twice as big). It would perhaps be interesting to plot the individual time-series lines themselves there (within a single cluster) next to the input time-series data of the same feature, then look for any correspondance.

For more understanding, I would recommend reading this great walkthrough/visualisation article covering t-SNE. There are a few examples that show non-circular results:

Wattenberg, et al., "How to Use t-SNE Effectively"

Answer 2

The lines therefore are the time series. t-sne is a very locally sensitive algorithm so every data point is very likely to show up nearest (in 2-D) to its nearest (N-D) neighbour.

In MNIST there is no sequence, so it doesn't show up as a line in 2-D. Every digit is written in a set of similar ways that cluster together. In your time-series data, if each time point is most similar to its next time point, what we expect is for them to be in some sequence - because you'd want point t=n to be closest to points t=n-1 and t=n+1 (if they are in fact the most similar).

Answer 3

I had a similar issue with lines created by T-SNE. In my case, it was due to not removing the index column (following numbers). The result was quite astonishing:

Answer 4

This is also likely to be due to the fact that you're using features from a softmax (or other probability) layer. Make sure you are using a layer of activations and not probabilities. For example, in Keras, if you extract features from the last layer, it extracts the softmax probability vector. Probability vectors are very similar to each other, especially if they belong to the same class, since this is what softmax tries to do. I have seen similar shapes using time-series data when extracting probabilities and then clouds of data (instead of curves) when extracting previous layer activations.