41
votes
Accepted
Intuitive explanation of Noise Contrastive Estimation (NCE) loss?
Taken from this post:https://stats.stackexchange.com/a/245452/154812
The issue
There are some issues with learning the word vectors using an "standard" neural network. In this way, the word vectors ...
- 534
24
votes
Accepted
One Hot Encoding vs Word Embedding - When to choose one or another?
One-Hot Encoding is a general method that can vectorize any categorical features. It is simple and fast to create and update the vectorization, just add a new entry in the vector with a one for each ...
- 19.5k
23
votes
How to initialize a new word2vec model with pre-trained model weights?
Thank Abhishek. I've figure it out! Here are my experiments.
1). we plot a easy example:
...
- 331
19
votes
How can I get a measure of the semantic similarity of words?
Word2vec does not capture similarity based on antonyms and synonyms. Word2vec would give a higher similarity if the two words have the similar context. Eg
The weather in California was _____ .
The ...
- 350
17
votes
Accepted
What is a better input for Word2Vec?
The answer to this question is that it depends. The primary approach is to pass in the tokenized sentences (so SentenceCorpus in your example), but depending on ...
- 391
17
votes
K-means clustering of word embedding gives strange results
Word embeddings are trained by substitutability, not similarity.
If you consider a sentence like "This food is unflavored." Then a good substitute word would be "flavored", and the sentence will ...
- 7,919
15
votes
What is the difference between and Embedding Layer and an Autoencoder?
Actually they are 3 different things (embedding layer, word2vec, autoencoder), though they can be used to solve similar problems. (i.e. dense representation of data)
Autoencoder is a type of neural ...
- 850
13
votes
Accepted
Predicting a word using Word2vec model
Word2vec works in two models CBOW and skip-gram. Let's take CBOW model, as your question goes in the same way that predict the target word, given the surrounding words.
Fundamentally, the model ...
- 1,944
13
votes
Accepted
doc2vec - How does the inference step work in PV-DBOW
The original paper does a lot of hand-waving on the implementation of inference step and is not clear. So your confusion is justified. I'll explain at high level below. I'm assuming only PV-DBOW model....
- 1,968
13
votes
BERT vs Word2VEC: Is bert disambiguating the meaning of the word vector?
BERT and ELMo are recent advances in the field. However, there is a fine but major distinction between them and the typical task of word-sense disambiguation: word2vec (and similar algorithms ...
- 181
13
votes
Accepted
Why is the cosine distance used to measure the similatiry between word embeddings?
You're asking two questions here.
Does this mean the magnitude of the vectors is irrelevant?
Yes. Cosine similarity is $ S_{cos} = \frac{A \cdot B}{\|A\|\|B\|} $, which just comes from the ...
- 1,214
12
votes
Accepted
Doc2Vec - How to label the paragraphs (gensim)
Both are possible. You can give every document a unique ID (such as a sequential serial number) as a doctag, or a shared string doctag representing something else about it, or both at the same time.
...
- 236
11
votes
Doc2Vec - How to label the paragraphs (gensim)
doc2vec model gets its algorithm from word2vec.
In word2vec there is no need to label the ...
- 1,944
11
votes
Accepted
Can we compare a word2vec vector with a doc2vec vector?
In paragraph vector, the vector tries to grasp the semantic meaning of all the words in the context by placing the vector itself in each and every context. Thus finally, the paragraph vector contains ...
- 1,944
11
votes
Accepted
How pre-trained BERT model generates word embeddings for out of vocabulary words?
BERT does not provide word-level representations, but subword representations. You may want to combine the vectors of all subwords of the same word (e.g. by averaging them), but that is up to you, ...
- 19.2k
10
votes
How word2vec can be used to identify unseen words and relate them to already trained data
Every algorithm that deals with text data has a vocabulary. In the case of word2vec, the vocabulary is comprised of all words in the input corpus, or at least those above the minimum-frequency ...
- 3,057
10
votes
How can I get a measure of the semantic similarity of words?
In Text Analytic Tools for Semantic Similarity, they developed a algorithm in order to find the similarity between 2 sentences. But if you read closely, they find the similarity of the word in a ...
- 2,296
10
votes
Accepted
Features of word vectors in Word2Vec
1- The number of features: In terms of neural network model it represents the number of neurons in the projection(hidden) layer. As the projection layer is built upon distributional hypothesis, ...
- 1,944
10
votes
How to initialize word-embeddings for Out of Vocabulary Word?
You have a few options here. Of these, I think 1 will be the easiest to implement, as it's a standard language model with an alignment term added to the loss. I'd recommend 2a if you think you have ...
- 568
10
votes
NN embedding layer
At theoretical level, the embedding layer is a linear layer, there is not any difference at all. However, in practice, if you are building a deep learning software, you have to make a difference ...
- 5,921
9
votes
How the embedding layer is trained in Keras Embedding layer
Both the answers are wrong.
An embedding layer is a trainable layer that contains 1 embedding matrix, which is two dimensional, in one axis the number of unique values the categorical input can take (...
- 9,258
9
votes
Why do we need 2 matrices for word2vec or GloVe
Might not be the answer you are seeking, but I'll still have a go:
First, quick review of word2Vec, assume we are using skip gram.
A typical Word2Vec train-able model consists of 1 input layer (for ...
- 2,676
9
votes
Accepted
Ratio between embedded vector dimensions and vocabulary size
The ratio of vocabulary vs embedding length to determine the size of other layers in a neural network doesn't really matter. Word embeddings are always around 100 and 300 in length, longer embedding ...
- 3,320
9
votes
Accepted
Difference between Gensim word2vec and keras Embedding layer
Yep, you're right! As you know, it's difficult for machine learning models to use natural language directly, so it helps to transform words into some meaningful numeric representation. This process ...
- 2,574
8
votes
Full Doc2Vec Implementation/Repdoduction in TensorFlow?
Implementations
These are all full scripts using Tensorflow, but just using Tensorflow does not ensure quality.
Bad
WangZ's implementation looks complete from a brief glance, although it only offers ...
8
votes
Training Doc2Vec and Word2Vec at the same time
You need to be careful with the assumptions you make about the doc2vec implementation. Here are some useful concepts:
Word2vec has two different model implementations (Skip-gram and Continuous-bag-of-...
- 1,812
8
votes
Ratio between embedded vector dimensions and vocabulary size
A similar question was asked here.
This Google Developers blog post says:
Well, the following "formula" provides a general rule of thumb about the number of embedding dimensions:
...
- 191
8
votes
Accepted
Text similarity with sentence embeddings
One approach is using Word Mover’s Distance (WMD). WMD is an algorithm for finding the distance between texts of different lengths, where each word is represented as a word embedding vector.
The ...
- 19.5k
8
votes
Accepted
How should I use BERT embeddings for clustering (as opposed to fine-tuning BERT model for a supervised task)
Here are the answers:
In sequence modeling, we expect a sentence to be ordered sequence, thus we cannot take random words (unlike bag of words, where we are just bothered about the words and not ...
- 949
Only top scored, non community-wiki answers of a minimum length are eligible