22
$\begingroup$

This is more like a general NLP question. What is the appropriate input to train a word embedding namely Word2Vec? Should all sentences belonging to an article be a separate document in a corpus? Or should each article be a document in said corpus? This is just an example using python and gensim.

Corpus split by sentence:

SentenceCorpus = [["first", "sentence", "of", "the", "first", "article."],
                  ["second", "sentence", "of", "the", "first", "article."],
                  ["first", "sentence", "of", "the", "second", "article."],
                  ["second", "sentence", "of", "the", "second", "article."]]

Corpus split by article:

ArticleCorpus = [["first", "sentence", "of", "the", "first", "article.",
                  "second", "sentence", "of", "the", "first", "article."],
                 ["first", "sentence", "of", "the", "second", "article.",
                  "second", "sentence", "of", "the", "second", "article."]]

Training Word2Vec in Python:

from gensim.models import Word2Vec

wikiWord2Vec = Word2Vec(ArticleCorpus)
$\endgroup$
14
$\begingroup$

The answer to this question is that it depends. The primary approach is to pass in the tokenized sentences (so SentenceCorpus in your example), but depending on what your goal is and what the corpus is you're looking at, you might want to instead use the entire article to learn the embeddings. This is something you might not know ahead of time -- so you'll have to think about how you want to evaluate the quality of the embeddings, and do some experiments to see which 'kind' of embeddings are more useful for your task(s).

$\endgroup$
  • $\begingroup$ Right on spot. I used the embeddings in a model and, just like you mentioned, there was a big improvement in the model's predictive performance when I used the entire article. So in what case would training sentence by sentence be superior. $\endgroup$ – wacax Dec 10 '15 at 0:11
  • 1
    $\begingroup$ You should look and see how words that have similar vectors are related to one another. There has been some work done on the size of the context window and type of context that suggests that smaller windows (and perhaps smaller document sizes, like sentences), might make words that are functionally similar (like US states) rather than topically similar (like US states and government-related words) have more similar vectors. I'm mostly citing Omer Levy and Yoav Goldberg's Dependency-Based Word Embeddings from 2014. I could be mistaken though and would love to be corrected if so. $\endgroup$ – NBartley Jan 21 '16 at 16:33
1
$\begingroup$

As a supplementary to @NBartley's answer. To anyone come across this question. I have tried use article/sentence as input for word2vec on Spark2.2, result as follow.

use sentence as input:

enter image description here

use article as input:

enter image description here

$\endgroup$
0
$\begingroup$

For the former, gensim has the Word2Vec class. For the latter, Doc2Vec.

http://rare-technologies.com/doc2vec-tutorial/

$\endgroup$
  • 3
    $\begingroup$ doc2vec is substantially different than performing word2vec on a corpus of articles rather than sentences. doc2vec will learn representations of the articles themselves, rather than just the words. $\endgroup$ – jamesmf Dec 9 '15 at 15:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.