I have used tsne to visualise a large dataset and it has produced the following graph. I need help interpreting it, as I have never seen a tsne graph like this before!
I am aware that not much overlapping data is a good thing, but I have rarely seen more than a few clusters. Has something gone wrong here? Or are there just a lot of clusters in this dataset.
Thanks. P
t-SNE is notoriously fickle. I would advise you read this short article How to Use t-SNE Effectively before you invest too much time trying to interpret the plots. In case you don't read it, you'll find that your t-SNE plot will change considerably based on your iterations, and perplexity. You'll find that distance between clusters aren't accurate and that cluster sizes aren't accurate too. t-SNE is a cool data reduction too, but often takes multiple runs to get a "good" plot. Laurens van der Maatan even suggest that running multiple iterations and picking the one with the lowest KL-divergence is perfectly okay. This is due to the fact that t-SNE is non-convex so you won't necessarily find a global minimum.
I had a similar problem, even more extreme looking. The data is from a depression screening survey as part of the Health and Nutrition Examination Survey. It's from "2017 - March 2020 Pre-pandemic" --> "Questionnaire Data" --> "Mental Health - Depression Screener". It has eleven columns, presumably one per question, and ~8k rows. Dropping NaN's, ~5k rows. I tried filling NaN's with mean, but the same plot occurred.
The fix wasn't the perplexity, but rather an index column was in the data. After dropping,
No clear clusters, but looks a lot better.
