I'd like to compare the difference among the same word mentioned in different sources. That is, how authors differ in their usage of ill-defined words, such as "democracy".

A brief plan was

  1. Take the books mentioning the term "democracy" as plain text
  2. In each book, replace democracy with democracy_%AuthorName%
  3. Train a word2vec model on these books
  4. Calculate the distance between democracy_AuthorA, democracy_AuthorB, and other relabeled mentions of "democracy"

So each author's "democracy" gets its own vector, which is used for comparison.

But it seems that word2vec requires much more than several books (each relabeled word occurs only in a subset of books) to train reliable vectors. The official page recommends datasets including billions of words.

I just wanted to ask how large should be the subset of one author's books to make such inference with word2vec or alternative tools, if available?

  • 1
    $\begingroup$ Are the books you are using solely on the topic of democracy, if not, might not your distance metric get swamped by larger differences between the books contents? This is a side effect of your problem being in a very high dimensional space and being touched by the hand of the curse of dimensionality. Perhaps taking only a small region of text around the word of interest would help, but it is still a problem with significant dimension. $\endgroup$ – image_doctor Jul 23 '15 at 6:41
  • 1
    $\begingroup$ Yes that's the essence of that. here goes with a probably ill thought out metaphor. Imagine chapters of books being represented by colours. And a book a a whole represented as the mixture of all the colours of the chapters. A book on democracy in western europe would likely end up with an overall reddish hue as the sum of it's chapters. If we represent tourism by blue, a book on Tourism in Cuba, with a sole chapter on democracy and it's influence on economic development, would have a strong blue hue. So the two books would appear very different when viewed as a whole. $\endgroup$ – image_doctor Jul 23 '15 at 8:38
  • 1
    $\begingroup$ That's the more accessible way of saying what a data scientist would phrase as the vectors for the two books will be a long way apart in feature space and so will appear quite dissimilar. It's really hard to quantify beforehand how many examples you will need without playing with the data, but language is subtle and layered so you will probably want as many as you can get .... and maybe more. Ultimately you won't know until you try. It's not a concrete answer, but unless someone direct experience of doing a similar thing, it's probably the best you will get. $\endgroup$ – image_doctor Jul 23 '15 at 8:43
  • 1
    $\begingroup$ word2vec already only uses "a small region of text around the word of interest." The window parameter sets how many words in the context are used to train the model for your word w $\endgroup$ – jamesmf Sep 9 '15 at 15:03
  • 1
    $\begingroup$ @politicalscientist I had not finished this project. $\endgroup$ – Anton Tarasenko Jan 9 '20 at 9:10

It sounds like Doc2Vec (or paragraph/context vectors) might be the right fit for this problem.

In a nutshell, in addition to the word vectors, you add a "context vector" (in your case, an embedding for the author) that is used to predict the center or context words.

This means that you would benefit from all the data about "democracy" but also extract an embedding for that author, which combined should allow you to analyze the bias of each author with limited data about each author.

You can use gensim's implementation. The doc includes links to the source papers.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.