Skip to main content
Notice removed Draw attention by Ingolifs
Bounty Ended with Brian Spiering's answer chosen by Ingolifs
Added further thoughts
Source Link
Ingolifs
  • 278
  • 1
  • 11

I've been training a word2vec/doc2vec model on a large amount of text. I recently stumbled across the t-SNE package, and am finding it wonderful at finding hidden structure in high-dimensional data.

Can t-SNE be used as a way of tracking the progress of a hard machine learning task like this - where the model's understanding goes from unintelligible nonsense to something with hidden structure?

I have seen examples of the MNIST data set on t-SNE where all the individual numbers cluster well with each other. (as explained in this answer) enter image description here

As I increase the number of vectors in the doc2vec model and the size of the training set, I start to see clumping (if you squint) in the t-SNE plot. So far, these clumps are mainly associated with posts of very similar wording (one clump is mainly "Good morning/evening!" tweets). (Picture was generated with perplexity of 400)

enter image description here

How much additional clumping can I expect to see as the model is improved? Is this indicative that the model is, in fact, improving and learning deeper connections between words/phrases? Or have these t-SNE plots settled into the form they'll always take?

EDIT: I have since realised that the apparent lack of clumping could be due to the data itself. MNIST separates out cleanly because there are generally no weird glyphs that look like halfway mutations between numbers. My dataset (twitter sentiment, 1.6 million tweets) is, for a lack of a better word, filled with unclassifiable drivel, and it seems entirely probable that the homogeneous forest of points in the centre of the plot represents these sorts of tweets.

I've been training a word2vec/doc2vec model on a large amount of text. I recently stumbled across the t-SNE package, and am finding it wonderful at finding hidden structure in high-dimensional data.

Can t-SNE be used as a way of tracking the progress of a hard machine learning task like this - where the model's understanding goes from unintelligible nonsense to something with hidden structure?

I have seen examples of the MNIST data set on t-SNE where all the individual numbers cluster well with each other. (as explained in this answer) enter image description here

As I increase the number of vectors in the doc2vec model and the size of the training set, I start to see clumping (if you squint) in the t-SNE plot. So far, these clumps are mainly associated with posts of very similar wording (one clump is mainly "Good morning/evening!" tweets). (Picture was generated with perplexity of 400)

enter image description here

How much additional clumping can I expect to see as the model is improved? Is this indicative that the model is, in fact, improving and learning deeper connections between words/phrases? Or have these t-SNE plots settled into the form they'll always take?

I've been training a word2vec/doc2vec model on a large amount of text. I recently stumbled across the t-SNE package, and am finding it wonderful at finding hidden structure in high-dimensional data.

Can t-SNE be used as a way of tracking the progress of a hard machine learning task like this - where the model's understanding goes from unintelligible nonsense to something with hidden structure?

I have seen examples of the MNIST data set on t-SNE where all the individual numbers cluster well with each other. (as explained in this answer) enter image description here

As I increase the number of vectors in the doc2vec model and the size of the training set, I start to see clumping (if you squint) in the t-SNE plot. So far, these clumps are mainly associated with posts of very similar wording (one clump is mainly "Good morning/evening!" tweets). (Picture was generated with perplexity of 400)

enter image description here

How much additional clumping can I expect to see as the model is improved? Is this indicative that the model is, in fact, improving and learning deeper connections between words/phrases? Or have these t-SNE plots settled into the form they'll always take?

EDIT: I have since realised that the apparent lack of clumping could be due to the data itself. MNIST separates out cleanly because there are generally no weird glyphs that look like halfway mutations between numbers. My dataset (twitter sentiment, 1.6 million tweets) is, for a lack of a better word, filled with unclassifiable drivel, and it seems entirely probable that the homogeneous forest of points in the centre of the plot represents these sorts of tweets.

Notice added Draw attention by Ingolifs
Bounty Started worth 50 reputation by Ingolifs
added 47 characters in body
Source Link
Ingolifs
  • 278
  • 1
  • 11

I've been training a word2vec/doc2vec model on a large amount of text. I recently stumbled across the t-SNE package, and am finding it wonderful at finding hidden structure in high-dimensional data.

Can t-SNE be used as a way of tracking the progress of a hard machine learning task like this - where the model's understanding goes from unintelligible nonsense to something with hidden structure?

I have seen examples of the MNIST data set on t-SNE where all the individual numbers cluster well with each other. (as explained in this answer) enter image description here

As I increase the number of vectors in the doc2vec model and the size of the training set, I start to see clumping (if you squint) in the t-SNE plot. So far, these clumps are mainly associated with posts of very similar wording (one clump is mainly "Good morning/evening!" tweets). (Picture was generated with perplexity of 400)

enter image description here

How much additional clumping can I expect to see as the model is improved? Is this indicative that the model is, in fact, improving and learning deeper connections between words/phrases? Or have these t-SNE plots settled into the form they'll always take?

I've been training a word2vec/doc2vec model on a large amount of text. I recently stumbled across the t-SNE package, and am finding it wonderful at finding hidden structure in high-dimensional data.

Can t-SNE be used as a way of tracking the progress of a hard machine learning task like this - where the model's understanding goes from unintelligible nonsense to something with hidden structure?

I have seen examples of the MNIST data set on t-SNE where all the individual numbers cluster well with each other. (as explained in this answer) enter image description here

As I increase the number of vectors in the doc2vec model and the size of the training set, I start to see clumping (if you squint) in the t-SNE plot. So far, these clumps are mainly associated with posts of very similar wording (one clump is mainly "Good morning/evening!" tweets).

enter image description here

How much additional clumping can I expect to see as the model is improved? Is this indicative that the model is, in fact, improving and learning deeper connections between words/phrases? Or have these t-SNE plots settled into the form they'll always take?

I've been training a word2vec/doc2vec model on a large amount of text. I recently stumbled across the t-SNE package, and am finding it wonderful at finding hidden structure in high-dimensional data.

Can t-SNE be used as a way of tracking the progress of a hard machine learning task like this - where the model's understanding goes from unintelligible nonsense to something with hidden structure?

I have seen examples of the MNIST data set on t-SNE where all the individual numbers cluster well with each other. (as explained in this answer) enter image description here

As I increase the number of vectors in the doc2vec model and the size of the training set, I start to see clumping (if you squint) in the t-SNE plot. So far, these clumps are mainly associated with posts of very similar wording (one clump is mainly "Good morning/evening!" tweets). (Picture was generated with perplexity of 400)

enter image description here

How much additional clumping can I expect to see as the model is improved? Is this indicative that the model is, in fact, improving and learning deeper connections between words/phrases? Or have these t-SNE plots settled into the form they'll always take?

Source Link
Ingolifs
  • 278
  • 1
  • 11

Using t-SNE to track progress of a word vector embedding model. Pitfalls?

I've been training a word2vec/doc2vec model on a large amount of text. I recently stumbled across the t-SNE package, and am finding it wonderful at finding hidden structure in high-dimensional data.

Can t-SNE be used as a way of tracking the progress of a hard machine learning task like this - where the model's understanding goes from unintelligible nonsense to something with hidden structure?

I have seen examples of the MNIST data set on t-SNE where all the individual numbers cluster well with each other. (as explained in this answer) enter image description here

As I increase the number of vectors in the doc2vec model and the size of the training set, I start to see clumping (if you squint) in the t-SNE plot. So far, these clumps are mainly associated with posts of very similar wording (one clump is mainly "Good morning/evening!" tweets).

enter image description here

How much additional clumping can I expect to see as the model is improved? Is this indicative that the model is, in fact, improving and learning deeper connections between words/phrases? Or have these t-SNE plots settled into the form they'll always take?