40 votes
Accepted

Intuitive explanation of Noise Contrastive Estimation (NCE) loss?

Taken from this post:https://stats.stackexchange.com/a/245452/154812 The issue There are some issues with learning the word vectors using an "standard" neural network. In this way, the word vectors ...
23 votes

How to initialize a new word2vec model with pre-trained model weights?

Thank Abhishek. I've figure it out! Here are my experiments. 1). we plot a easy example: ...
23 votes
Accepted

One Hot Encoding vs Word Embedding - When to choose one or another?

One-Hot Encoding is a general method that can vectorize any categorical features. It is simple and fast to create and update the vectorization, just add a new entry in the vector with a one for each ...
19 votes

How can I get a measure of the semantic similarity of words?

Word2vec does not capture similarity based on antonyms and synonyms. Word2vec would give a higher similarity if the two words have the similar context. Eg The weather in California was _____ . The ...
17 votes
Accepted

What is a better input for Word2Vec?

The answer to this question is that it depends. The primary approach is to pass in the tokenized sentences (so SentenceCorpus in your example), but depending on ...
  • 391
17 votes

K-means clustering of word embedding gives strange results

Word embeddings are trained by substitutability, not similarity. If you consider a sentence like "This food is unflavored." Then a good substitute word would be "flavored", and the sentence will ...
13 votes
Accepted

Predicting a word using Word2vec model

Word2vec works in two models CBOW and skip-gram. Let's take CBOW model, as your question goes in the same way that predict the target word, given the surrounding words. Fundamentally, the model ...
  • 1,934
13 votes
Accepted

doc2vec - How does the inference step work in PV-DBOW

The original paper does a lot of hand-waving on the implementation of inference step and is not clear. So your confusion is justified. I'll explain at high level below. I'm assuming only PV-DBOW model....
  • 1,958
13 votes

What is the difference between and Embedding Layer and an Autoencoder?

Actually they are 3 different things (embedding layer, word2vec, autoencoder), though they can be used to solve similar problems. (i.e. dense representation of data) Autoencoder is a type of neural ...
  • 830
13 votes

BERT vs Word2VEC: Is bert disambiguating the meaning of the word vector?

BERT and ELMo are recent advances in the field. However, there is a fine but major distinction between them and the typical task of word-sense disambiguation: word2vec (and similar algorithms ...
12 votes
Accepted

Doc2Vec - How to label the paragraphs (gensim)

Both are possible. You can give every document a unique ID (such as a sequential serial number) as a doctag, or a shared string doctag representing something else about it, or both at the same time. ...
  • 236
11 votes

Doc2Vec - How to label the paragraphs (gensim)

doc2vec model gets its algorithm from word2vec. In word2vec there is no need to label the ...
  • 1,934
11 votes
Accepted

Can we compare a word2vec vector with a doc2vec vector?

In paragraph vector, the vector tries to grasp the semantic meaning of all the words in the context by placing the vector itself in each and every context. Thus finally, the paragraph vector contains ...
  • 1,934
11 votes
Accepted

Why is the cosine distance used to measure the similatiry between word embeddings?

You're asking two questions here. Does this mean the magnitude of the vectors is irrelevant? Yes. Cosine distance is $ D_{cos} = \frac{A \cdot B}{\|A\|\|B\|} $, which just comes from the definition ...
  • 1,184
10 votes

How word2vec can be used to identify unseen words and relate them to already trained data

Every algorithm that deals with text data has a vocabulary. In the case of word2vec, the vocabulary is comprised of all words in the input corpus, or at least those above the minimum-frequency ...
  • 3,047
10 votes

How can I get a measure of the semantic similarity of words?

In Text Analytic Tools for Semantic Similarity, they developed a algorithm in order to find the similarity between 2 sentences. But if you read closely, they find the similarity of the word in a ...
  • 2,296
10 votes
Accepted

Features of word vectors in Word2Vec

1- The number of features: In terms of neural network model it represents the number of neurons in the projection(hidden) layer. As the projection layer is built upon distributional hypothesis, ...
  • 1,934
10 votes

How to initialize word-embeddings for Out of Vocabulary Word?

You have a few options here. Of these, I think 1 will be the easiest to implement, as it's a standard language model with an alignment term added to the loss. I'd recommend 2a if you think you have ...
10 votes

NN embedding layer

At theoretical level, the embedding layer is a linear layer, there is not any difference at all. However, in practice, if you are building a deep learning software, you have to make a difference ...
  • 5,859
10 votes
Accepted

How pre-trained BERT model generates word embeddings for out of vocabulary words?

BERT does not provide word-level representations, but subword representations. You may want to combine the vectors of all subwords of the same word (e.g. by averaging them), but that is up to you, ...
  • 16.9k
9 votes

How the embedding layer is trained in Keras Embedding layer

Both the answers are wrong. An embedding layer is a trainable layer that contains 1 embedding matrix, which is two dimensional, in one axis the number of unique values the categorical input can take (...
9 votes

Why do we need 2 matrices for word2vec or GloVe

Might not be the answer you are seeking, but I'll still have a go: First, quick review of word2Vec, assume we are using skip gram. A typical Word2Vec train-able model consists of 1 input layer (for ...
  • 2,626
9 votes
Accepted

Ratio between embedded vector dimensions and vocabulary size

The ratio of vocabulary vs embedding length to determine the size of other layers in a neural network doesn't really matter. Word embeddings are always around 100 and 300 in length, longer embedding ...
  • 3,310
9 votes
Accepted

Difference between Gensim word2vec and keras Embedding layer

Yep, you're right! As you know, it's difficult for machine learning models to use natural language directly, so it helps to transform words into some meaningful numeric representation. This process ...
  • 2,464
8 votes

Full Doc2Vec Implementation/Repdoduction in TensorFlow?

Implementations These are all full scripts using Tensorflow, but just using Tensorflow does not ensure quality. Bad WangZ's implementation looks complete from a brief glance, although it only offers ...
8 votes

Training Doc2Vec and Word2Vec at the same time

You need to be careful with the assumptions you make about the doc2vec implementation. Here are some useful concepts: Word2vec has two different model implementations (Skip-gram and Continuous-bag-of-...
  • 1,772
8 votes

Ratio between embedded vector dimensions and vocabulary size

A similar question was asked here. This Google Developers blog post says: Well, the following "formula" provides a general rule of thumb about the number of embedding dimensions: ...
  • 191
8 votes
Accepted

How should I use BERT embeddings for clustering (as opposed to fine-tuning BERT model for a supervised task)

Here are the answers: In sequence modeling, we expect a sentence to be ordered sequence, thus we cannot take random words (unlike bag of words, where we are just bothered about the words and not ...
8 votes
Accepted

Which method is more suitable? overfitting of traning data or low accuracy?

The performance on in-sample data almost does not count. The performance on out-of-sample data is more indicative of how you should expect your model to perform on future inputs. The second model has ...
  • 3,581

Only top scored, non community-wiki answers of a minimum length are eligible