2
$\begingroup$

I've been training a word2vec/doc2vec model on a large amount of text. I recently stumbled across the t-SNE package, and am finding it wonderful at finding hidden structure in high-dimensional data.

Can t-SNE be used as a way of tracking the progress of a hard machine learning task like this - where the model's understanding goes from unintelligible nonsense to something with hidden structure?

I have seen examples of the MNIST data set on t-SNE where all the individual numbers cluster well with each other. (as explained in this answer) enter image description here

As I increase the number of vectors in the doc2vec model and the size of the training set, I start to see clumping (if you squint) in the t-SNE plot. So far, these clumps are mainly associated with posts of very similar wording (one clump is mainly "Good morning/evening!" tweets). (Picture was generated with perplexity of 400)

enter image description here

How much additional clumping can I expect to see as the model is improved? Is this indicative that the model is, in fact, improving and learning deeper connections between words/phrases? Or have these t-SNE plots settled into the form they'll always take?

EDIT: I have since realised that the apparent lack of clumping could be due to the data itself. MNIST separates out cleanly because there are generally no weird glyphs that look like halfway mutations between numbers. My dataset (twitter sentiment, 1.6 million tweets) is, for a lack of a better word, filled with unclassifiable drivel, and it seems entirely probable that the homogeneous forest of points in the centre of the plot represents these sorts of tweets.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
+50
$\begingroup$

T-SNE is extremely useful for visualizing high-dimensional data in lower-dimensional space. However, t-SNE can have several gotchas, including comparing cluster sizes. The t-sne algorithm tries to even out cluster sizes by expanding dense clusters and contracting sparse clusters. Thus, it is not straightforward to directly compare clusters across different runs.

"How to Use t-SNE Effectively" goes into greater detail about common pitfalls of using the technique.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Okay, it's helpful to know that t-SNE 'cheats' a little bit by making the clusters more visually apparent. It also seems that due to the high likelihood of finding oneself in a local minima, one can't gain information on incremental changes just by "Squintin' and Reckonin'" Even so, I was hoping for more advice from someone experienced with this particular sort of problem (training word vectors) on whether I could use this to judge the progress of a learning task, and how much weight I ought to give to t-SNE analysis. $\endgroup$
    – Ingolifs
    Commented Feb 4, 2019 at 1:18
  • 1
    $\begingroup$ The short answer is no - don't use t-SNE to judge the progress of a learning word embedding. It is better to test custom embedding against established embeddings aclweb.org/aclwiki/Google_analogy_test_set_(State_of_the_art) $\endgroup$
    – Brian Spiering
    Commented Feb 4, 2019 at 2:00
1
$\begingroup$

tSNE has nothing to do with model or classification it is just a projection of high dimensional data points in 2d plot and nothing else, but it just gives you a hint if the clumps are well separated then your classifier would do better but not always true, because every classifier model has different optimization technique and mathematical formulation e.g. from decision trees to neural nets, however, that is another thing they may arrive at the same solution.
Because finally every classifier has to do one thing draw a line or a hyper-plane that separates different classes of data points.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.