43
votes
Accepted
Intuitive explanation of Noise Contrastive Estimation (NCE) loss?
Taken from this post:https://stats.stackexchange.com/a/245452/154812
The issue
There are some issues with learning the word vectors using a "standard" neural network. In this way, the word ...
- 554
28
votes
Accepted
One Hot Encoding vs Word Embedding - When to choose one or another?
One-Hot Encoding is a general method that can vectorize any categorical features. It is simple and fast to create and update the vectorization, just add a new entry in the vector with a one for each ...
- 22.3k
23
votes
How to initialize a new word2vec model with pre-trained model weights?
Thank Abhishek. I've figure it out! Here are my experiments.
1). we plot a easy example:
...
- 331
19
votes
How can I get a measure of the semantic similarity of words?
Word2vec does not capture similarity based on antonyms and synonyms. Word2vec would give a higher similarity if the two words have the similar context. Eg
The weather in California was _____ .
The ...
- 350
17
votes
K-means clustering of word embedding gives strange results
Word embeddings are trained by substitutability, not similarity.
If you consider a sentence like "This food is unflavored." Then a good substitute word would be "flavored", and the sentence will ...
- 8,114
16
votes
Accepted
Why is the cosine distance used to measure the similatiry between word embeddings?
You're asking two questions here.
Does this mean the magnitude of the vectors is irrelevant?
Yes. Cosine similarity is $ S_{cos} = \frac{A \cdot B}{\|A\|\|B\|} $, which just comes from the ...
- 1,294
15
votes
What is the difference between and Embedding Layer and an Autoencoder?
Actually they are 3 different things (embedding layer, word2vec, autoencoder), though they can be used to solve similar problems. (i.e. dense representation of data)
Autoencoder is a type of neural ...
- 860
14
votes
Accepted
doc2vec - How does the inference step work in PV-DBOW
The original paper does a lot of hand-waving on the implementation of inference step and is not clear. So your confusion is justified. I'll explain at high level below. I'm assuming only PV-DBOW model....
- 2,018
14
votes
BERT vs Word2VEC: Is bert disambiguating the meaning of the word vector?
BERT and ELMo are recent advances in the field. However, there is a fine but major distinction between them and the typical task of word-sense disambiguation: word2vec (and similar algorithms ...
- 191
14
votes
Accepted
How pre-trained BERT model generates word embeddings for out of vocabulary words?
BERT does not provide word-level representations, but subword representations. You may want to combine the vectors of all subwords of the same word (e.g. by averaging them), but that is up to you, ...
- 27.5k
10
votes
How to initialize word-embeddings for Out of Vocabulary Word?
You have a few options here. Of these, I think 1 will be the easiest to implement, as it's a standard language model with an alignment term added to the loss. I'd recommend 2a if you think you have ...
- 578
10
votes
Accepted
Ratio between embedded vector dimensions and vocabulary size
The ratio of vocabulary vs embedding length to determine the size of other layers in a neural network doesn't really matter. Word embeddings are always around 100 and 300 in length, longer embedding ...
- 3,460
10
votes
NN embedding layer
At theoretical level, the embedding layer is a linear layer, there is not any difference at all. However, in practice, if you are building a deep learning software, you have to make a difference ...
- 6,106
9
votes
How the embedding layer is trained in Keras Embedding layer
Both the answers are wrong.
An embedding layer is a trainable layer that contains 1 embedding matrix, which is two dimensional, in one axis the number of unique values the categorical input can take (...
- 9,398
9
votes
Why do we need 2 matrices for word2vec or GloVe
Might not be the answer you are seeking, but I'll still have a go:
First, quick review of word2Vec, assume we are using skip gram.
A typical Word2Vec train-able model consists of 1 input layer (for ...
- 2,756
9
votes
Accepted
Text similarity with sentence embeddings
One approach is using Word Mover’s Distance (WMD). WMD is an algorithm for finding the distance between texts of different lengths, where each word is represented as a word embedding vector.
The ...
- 22.3k
9
votes
Accepted
Difference between Gensim word2vec and keras Embedding layer
Yep, you're right! As you know, it's difficult for machine learning models to use natural language directly, so it helps to transform words into some meaningful numeric representation. This process ...
- 2,772
9
votes
Accepted
How should I use BERT embeddings for clustering (as opposed to fine-tuning BERT model for a supervised task)
Here are the answers:
In sequence modeling, we expect a sentence to be ordered sequence, thus we cannot take random words (unlike bag of words, where we are just bothered about the words and not ...
- 1,197
8
votes
Full Doc2Vec Implementation/Repdoduction in TensorFlow?
Implementations
These are all full scripts using Tensorflow, but just using Tensorflow does not ensure quality.
Bad
WangZ's implementation looks complete from a brief glance, although it only offers ...
8
votes
Training Doc2Vec and Word2Vec at the same time
You need to be careful with the assumptions you make about the doc2vec implementation. Here are some useful concepts:
Word2vec has two different model implementations (Skip-gram and Continuous-bag-of-...
- 1,882
8
votes
Ratio between embedded vector dimensions and vocabulary size
A similar question was asked here.
This Google Developers blog post says:
Well, the following "formula" provides a general rule of thumb about the number of embedding dimensions:
...
- 201
8
votes
Accepted
Backpropgating error to emedding matrix
An embedding layer is in fact a linear layer. It maps the input, using a matrix multiplication, to the output, without any activation function after the multiplication. Therefore, the backpropagation ...
- 6,106
8
votes
Accepted
Which method is more suitable? overfitting of traning data or low accuracy?
The performance on in-sample data almost does not count. The performance on out-of-sample data is more indicative of how you should expect your model to perform on future inputs.
The second model has ...
- 4,244
7
votes
How to overcome training example's different lengths when working with Word Embeddings (word2vec)
Let me suggest three simple options:
average the vectors (component-wise), i.e., compute the word embedding vector for each word in the text, and average them. (as suggested by others).
take the (...
- 3,401
7
votes
Reducing the dimensionality of word embeddings
There is a paper on this subject called
Simple and Effective Dimensionality Reduction for Word Embeddings, Vikas Raunak
You can read it here
You can also find the implementation here
In my opinion ...
- 171
7
votes
Why do we need 2 matrices for word2vec or GloVe
why we actually need two matrices (and not one) for these models. Couldn't we use
the same one for U and V?
In principle, you are right, we can. But we don't want to, since the increase in the ...
- 9,442
7
votes
Accepted
What is the vector value of [CLS] [SEP] tokens in BERT
First a clarification: there is no masking at all in the [CLS] and [SEP] tokens. These are artificial tokens that are ...
- 27.5k
7
votes
Accepted
Why using a frozen embedding layer in an LSTM model
The embedding matrix which used in the initialization of the Embedding layer is highly trained on a large corpus of text. The training and the data are so huge that ...
- 2,220
7
votes
BERT vs Word2VEC: Is bert disambiguating the meaning of the word vector?
I think there are a few misconceptions in your statements. Please take into account the following
BERT does not provide word-level representation. It provides sub-words embeddings and sentence ...
- 27.5k
7
votes
Similarity of words using BERTMODEL
First of all, I think you are confused with pretrained and finetuned.
BERT is pretrained on ...
- 1,014
Only top scored, non community-wiki answers of a minimum length are eligible