1
$\begingroup$

I have created a Word2Vec model based on the transcript of the Office. I am now trying to visualize the embedding space for the top similar words of an input word with t-SNE in 2D and 3D. I additionally color the similarity value of a word to the input word (colored in red). When I plot the t-SNE output the similar words seem to be clustered/laying around near the input word (paper in this case), which makes sense. However, when I use t-SNE again for a 3D plot the similarity is not as well visible anymore.

2D t-SNE visualization

3D t-SNE visualization

Why are the words that are close in the 2D plot and of high similarity not also very close in the 3D plot?

Thanks already for the help!

$\endgroup$

2 Answers 2

0
$\begingroup$

How many iterations did you applied?

Normally, the end result for 2D and 3D should be quite similar in terms of cluster groups.

If there are many differences, it could be explained by different factors:

  • The data is not enough differentiable. In this case, you would need more data to make a better generalization.
  • There are not enough iterations during training. t-SNE often requires thousands of iterations to reach a good balance.
  • Subcategories could require starting t-SNE again. Even if the t-SNE model is well trained, it is designed to differentiate in main clusters, but sub-clusters may not be well differentiated. In this case, if there is a "zoom" around a word, restarting t-SNE should be necessary. It is something done already in the TensorFlow Projector (select a point, isolate and start t-SNE).
  • The initialization is not random enough. You could set a random seed to make the results more reproducible.
  • The perplexity is too high. In many cases, a low perplexity (around 5) could be necessary to make clear clusters.

According to your screenshots, I think the first or the third options are the most possible ones.

Note: UMAP is also interesting because it has a logic between clusters, in addition to good clusterizing.

$\endgroup$
1
  • 1
    $\begingroup$ Thank you for your help! I already got better results by lowering the perplexity to 2 and by increasing the number of iterations to 7000. I'm aware now of the other points and will consider them for later cases. $\endgroup$
    – Elodin
    Commented Dec 15, 2022 at 8:32
0
$\begingroup$

Those results are to be expected because clustering t-SNE (t-distributed stochastic neighbor embedding) data is known to not be interable because t-SNE adapts its notion of “distance” to regional density variations in the data set.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.