Skip to main content
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more
42 votes
Accepted

Intuitive explanation of Noise Contrastive Estimation (NCE) loss?

Taken from this post:https://stats.stackexchange.com/a/245452/154812 The issue There are some issues with learning the word vectors using a "standard" neural network. In this way, the word ...
user154812's user avatar
28 votes
Accepted

One Hot Encoding vs Word Embedding - When to choose one or another?

One-Hot Encoding is a general method that can vectorize any categorical features. It is simple and fast to create and update the vectorization, just add a new entry in the vector with a one for each ...
Brian Spiering's user avatar
23 votes

How to initialize a new word2vec model with pre-trained model weights?

Thank Abhishek. I've figure it out! Here are my experiments. 1). we plot a easy example: ...
Shixiang Wan's user avatar
19 votes

How can I get a measure of the semantic similarity of words?

Word2vec does not capture similarity based on antonyms and synonyms. Word2vec would give a higher similarity if the two words have the similar context. Eg The weather in California was _____ . The ...
Trideep Rath's user avatar
17 votes

K-means clustering of word embedding gives strange results

Word embeddings are trained by substitutability, not similarity. If you consider a sentence like "This food is unflavored." Then a good substitute word would be "flavored", and the sentence will ...
Has QUIT--Anony-Mousse's user avatar
16 votes
Accepted

Why is the cosine distance used to measure the similatiry between word embeddings?

You're asking two questions here. Does this mean the magnitude of the vectors is irrelevant? Yes. Cosine similarity is $ S_{cos} = \frac{A \cdot B}{\|A\|\|B\|} $, which just comes from the ...
Matthew's user avatar
  • 1,284
15 votes

What is the difference between and Embedding Layer and an Autoencoder?

Actually they are 3 different things (embedding layer, word2vec, autoencoder), though they can be used to solve similar problems. (i.e. dense representation of data) Autoencoder is a type of neural ...
Viktor's user avatar
  • 850
14 votes

BERT vs Word2VEC: Is bert disambiguating the meaning of the word vector?

BERT and ELMo are recent advances in the field. However, there is a fine but major distinction between them and the typical task of word-sense disambiguation: word2vec (and similar algorithms ...
Juanvulcano's user avatar
14 votes
Accepted

How pre-trained BERT model generates word embeddings for out of vocabulary words?

BERT does not provide word-level representations, but subword representations. You may want to combine the vectors of all subwords of the same word (e.g. by averaging them), but that is up to you, ...
noe's user avatar
  • 26.9k
13 votes
Accepted

doc2vec - How does the inference step work in PV-DBOW

The original paper does a lot of hand-waving on the implementation of inference step and is not clear. So your confusion is justified. I'll explain at high level below. I'm assuming only PV-DBOW model....
hssay's user avatar
  • 1,998
11 votes
Accepted

Can we compare a word2vec vector with a doc2vec vector?

In paragraph vector, the vector tries to grasp the semantic meaning of all the words in the context by placing the vector itself in each and every context. Thus finally, the paragraph vector contains ...
chmodsss's user avatar
  • 1,964
10 votes

How can I get a measure of the semantic similarity of words?

In Text Analytic Tools for Semantic Similarity, they developed a algorithm in order to find the similarity between 2 sentences. But if you read closely, they find the similarity of the word in a ...
Hima Varsha's user avatar
  • 2,346
10 votes
Accepted

Features of word vectors in Word2Vec

1- The number of features: In terms of neural network model it represents the number of neurons in the projection(hidden) layer. As the projection layer is built upon distributional hypothesis, ...
chmodsss's user avatar
  • 1,964
10 votes

How to initialize word-embeddings for Out of Vocabulary Word?

You have a few options here. Of these, I think 1 will be the easiest to implement, as it's a standard language model with an alignment term added to the loss. I'd recommend 2a if you think you have ...
Nix Searcy's user avatar
10 votes
Accepted

Ratio between embedded vector dimensions and vocabulary size

The ratio of vocabulary vs embedding length to determine the size of other layers in a neural network doesn't really matter. Word embeddings are always around 100 and 300 in length, longer embedding ...
wacax's user avatar
  • 3,400
10 votes

NN embedding layer

At theoretical level, the embedding layer is a linear layer, there is not any difference at all. However, in practice, if you are building a deep learning software, you have to make a difference ...
David Masip's user avatar
  • 6,081
9 votes

How the embedding layer is trained in Keras Embedding layer

Both the answers are wrong. An embedding layer is a trainable layer that contains 1 embedding matrix, which is two dimensional, in one axis the number of unique values the categorical input can take (...
Jan van der Vegt's user avatar
9 votes

Why do we need 2 matrices for word2vec or GloVe

Might not be the answer you are seeking, but I'll still have a go: First, quick review of word2Vec, assume we are using skip gram. A typical Word2Vec train-able model consists of 1 input layer (for ...
Kari's user avatar
  • 2,726
9 votes
Accepted

Difference between Gensim word2vec and keras Embedding layer

Yep, you're right! As you know, it's difficult for machine learning models to use natural language directly, so it helps to transform words into some meaningful numeric representation. This process ...
zachdj's user avatar
  • 2,724
9 votes
Accepted

How should I use BERT embeddings for clustering (as opposed to fine-tuning BERT model for a supervised task)

Here are the answers: In sequence modeling, we expect a sentence to be ordered sequence, thus we cannot take random words (unlike bag of words, where we are just bothered about the words and not ...
Ashwin Geet D'Sa's user avatar
8 votes

Full Doc2Vec Implementation/Repdoduction in TensorFlow?

Implementations These are all full scripts using Tensorflow, but just using Tensorflow does not ensure quality. Bad WangZ's implementation looks complete from a brief glance, although it only offers ...
Nicholas Roth's user avatar
8 votes

Training Doc2Vec and Word2Vec at the same time

You need to be careful with the assumptions you make about the doc2vec implementation. Here are some useful concepts: Word2vec has two different model implementations (Skip-gram and Continuous-bag-of-...
TitoOrt's user avatar
  • 1,872
8 votes

Ratio between embedded vector dimensions and vocabulary size

A similar question was asked here. This Google Developers blog post says: Well, the following "formula" provides a general rule of thumb about the number of embedding dimensions: ...
Tom Hale's user avatar
  • 201
8 votes
Accepted

Backpropgating error to emedding matrix

An embedding layer is in fact a linear layer. It maps the input, using a matrix multiplication, to the output, without any activation function after the multiplication. Therefore, the backpropagation ...
David Masip's user avatar
  • 6,081
8 votes
Accepted

Text similarity with sentence embeddings

One approach is using Word Mover’s Distance (WMD). WMD is an algorithm for finding the distance between texts of different lengths, where each word is represented as a word embedding vector. The ...
Brian Spiering's user avatar
8 votes
Accepted

Which method is more suitable? overfitting of traning data or low accuracy?

The performance on in-sample data almost does not count. The performance on out-of-sample data is more indicative of how you should expect your model to perform on future inputs. The second model has ...
Dave's user avatar
  • 3,903
7 votes

Reducing the dimensionality of word embeddings

There is a paper on this subject called Simple and Effective Dimensionality Reduction for Word Embeddings, Vikas Raunak You can read it here You can also find the implementation here In my opinion ...
Gabriel M's user avatar
  • 171
7 votes

How to overcome training example's different lengths when working with Word Embeddings (word2vec)

Let me suggest three simple options: average the vectors (component-wise), i.e., compute the word embedding vector for each word in the text, and average them. (as suggested by others). take the (...
D.W.'s user avatar
  • 3,371
7 votes

Why do we need 2 matrices for word2vec or GloVe

why we actually need two matrices (and not one) for these models. Couldn't we use the same one for U and V? In principle, you are right, we can. But we don't want to, since the increase in the ...
Esmailian's user avatar
  • 9,322
7 votes
Accepted

Why using a frozen embedding layer in an LSTM model

The embedding matrix which used in the initialization of the Embedding layer is highly trained on a large corpus of text. The training and the data are so huge that ...
Shubham Panchal's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible