In the process of understanding how Word2Vec in Spark differs from gensim one, I got very confused by the example presented in the Spark docs (reference link: https://spark.apache.org/docs/2.2.0/ml-features.html#word2vec) and I was wondering why instead of transforming single words, they transform entire sentences. Isn't word2vec purpose to embed single words into a vector space? Why they embed entire sentences? How can one properly train word2vec and then applying to single words in Spark?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
Spark (naively) uses average of vectors for all words in the document as representation of the document. Check the API documentation little carefully.
"The Word2VecModel transforms each document into a vector using the average of all words in the document ; this vector can then be used as features for prediction, document similarity calculations, etc."
If you are interested specifically in vector for a word (and not document), you can check getVectors method which will return data-frame of word and vector. The API behaviour does give rise to confusion though as it assumes that everyone wants to use averaging by default.
-
$\begingroup$ I just noticed that while getVectors is available in pyspark, it is not present in Scala API. In that case, is the only way to get word vectors, "exploding" the input text and call transform on it? $\endgroup$– datapugCommented Dec 26, 2018 at 19:15
-
$\begingroup$ It does have getVectors in scala API for Word2VecModel. Please check - spark.apache.org/docs/2.2.0/api/scala/… $\endgroup$ Commented Feb 2, 2019 at 19:03