I am trying to do sentiment analysis. In order to convert the words to word vectors, I am using Word2Vec model. Suppose I have all the sentences in a list named 'sentences' and I am passing these sentences to word2vec as follows:

model = word2vec.Word2Vec(sentences, workers=4 , min_count=40, size=300,   window=5, sample=1e-3)

Since I am noob to word vectors, I have two doubts:

1- Setting the number of features to 300 defines the features of a word vector. But what these features signify? If each word in this model is represented by a 1x300 numpy array, then what do these 300 features signify for that word?

2- What does downsampling as represented by 'sample' parameter in the above model do in actual?


1- The number of features: In terms of neural network model it represents the number of neurons in the projection(hidden) layer. As the projection layer is built upon distributional hypothesis, numerical vector for each word signifies it's relation with its context words.

These features are learnt by the neural network as this is unsupervised method. Each vector has several set of semantic characteristics. For instance, let's take the classical example, V(King) -V(man) + V(Women) ~ V(Queen) and each word represented by 300-d vector. V(King) will have semantic characteristics of Royality, kingdom, masculinity, human in the vector in a certain order. V(man) will have masculinity, human, work in a certain order. Thus when V(King)-V(Man) is done, masculinity,human characteristics will get nullified and when added with V(Women) which having femininity, human characteristics will be added thus resulting in a vector much similar to V(Queen). The interesting thing is, these characteristics are encoded in the vector in a certain order so that numerical computations such as addition, subtraction works perfectly. This is due to the nature of unsupervised learning method in neural network.

2- There are two approximation algorithms. Hierarchical softmax and negative sampling. When the sample parameter is given, it takes negative sampling. In case of hierarchical softmax, for each word vector its context words are given positive outputs and all other words in vocabulary are given negative outputs. The issue of time complexity is resolved by negative sampling. As in negative sampling, rather than the whole vocabulary, only a sampled part of vocabulary is given negative outputs and the vectors are trained which is so much faster than former method.

  • 1
    $\begingroup$ This interpretation of word2vec features is misleading. There isn't a masculinity dimension of the space, or a royalty element in the vector. If that were the case, then a 300-dimension vector space could only represent 300 independent semantic dichotomies. $\endgroup$ – Dan Hicks Jul 9 '17 at 12:59
  • $\begingroup$ @DanHicks: I have never mentioned each feature as a dimension of space. I just told that such semantic features are encoded in the vector in certain order, such that mathematical operations are possible. $\endgroup$ – chmodsss Jul 9 '17 at 14:29
  • $\begingroup$ "Features" normally refers to the variables used to represent the cases — in this case, the elements of the word vectors/dimensions of the vector space. @Nain's question clearly uses "features" in this way. The "semantic features" you're talking about are at best a vague way of talking about how word2vec handles analogies. They are not at all the features of the word vectors. $\endgroup$ – Dan Hicks Jul 10 '17 at 0:12
  • 1
    $\begingroup$ you are right.. I have edited "semantic features" to "semantic characteristics", and the "features" in the answer represents only the dimensions of the vector. $\endgroup$ – chmodsss Jul 10 '17 at 7:45
  1. According to distributional hypothesis, individual dimension in the vector of the word does not signify much about the word in real world. You need to worry about the individual dimensions. If your question is so how should I select the number of dimensions, it is purely based on experiment for your data and it can go from 100 to 1000. For many experiments where the training is done on wiki text, the 300 dimension mostly give the best result.
  2. Sample param is the parameter used to prune the words having high frequency. E.g., "the", "is", "was", these stop words are not considered in the window while predicting the inside word and the default value works well to identify these stop words whose frequency is higher.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.