Questions tagged [word-embeddings]
Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.
507
questions
44
votes
5
answers
41k
views
Intuitive explanation of Noise Contrastive Estimation (NCE) loss?
I read about NCE (a form of candidate sampling) from these two sources:
Tensorflow writeup
Original Paper
Can someone help me with the following:
A simple explanation of how NCE works (I found the ...
- 4,025
33
votes
5
answers
40k
views
How can I get a measure of the semantic similarity of words?
What is the best way to figure out the semantic similarity of words? Word2Vec is okay, but not ideal:
...
- 665
30
votes
3
answers
20k
views
What is a better input for Word2Vec?
This is more like a general NLP question.
What is the appropriate input to train a word embedding namely Word2Vec? Should all sentences belonging to an article be a separate document in a corpus? Or ...
- 3,390
28
votes
2
answers
24k
views
Predicting a word using Word2vec model
Given a sentence:
"When I open the ?? door it starts heating automatically"
I would like to get the list of possible words in ?? with a probability.
The basic concept used in word2vec model ...
- 345
27
votes
5
answers
19k
views
BERT vs Word2VEC: Is bert disambiguating the meaning of the word vector?
Word2vec:
Word2vec provides a vector for each token/word and those vectors encode the meaning of the word. Although those vectors are not human interpretable, the meaning of the vectors are ...
- 521
22
votes
2
answers
18k
views
Doc2Vec - How to label the paragraphs (gensim)
I am wondering how to label (tag) sentences / paragraphs / documents with doc2vec in gensim - from a practical standpoint.
Do you need to have each sentence / paragraph / document with its own ...
- 702
21
votes
4
answers
44k
views
How to initialize a new word2vec model with pre-trained model weights?
I am using Gensim Library in python for using and training word2vector model. Recently, I was looking at initializing my model weights with some pre-trained word2vec model such as (GoogleNewDataset ...
- 461
16
votes
2
answers
30k
views
One Hot Encoding vs Word Embedding - When to choose one or another?
A colleague of mine is having an interesting situation, he has quite a large set of possibilities for a defined categorical feature (+/- 300 different values)
The usual data science approach would be ...
14
votes
4
answers
18k
views
How word2vec can be used to identify unseen words and relate them to already trained data
I was working on word2vec gensim model and found it really interesting.
I am intersted in finding how a unknown/unseen word when checked with the model will be able to get similar terms from the ...
- 341
13
votes
3
answers
5k
views
Why do we need 2 matrices for word2vec or GloVe
Word2vec and GloVe are the two most known words embedding methods. Many works pointed that these two models are actually very close to each other and that under some assumptions, they perform a matrix ...
- 1,307
12
votes
2
answers
10k
views
Shall I use the Euclidean Distance or the Cosine Similarity to compute the semantic similarity of two words?
I want to compute the semantic similarity of two words using their vector representations (obtained using e.g. word2vec, GloVe, etc.). Shall I use the Euclidean Distance or the Cosine Similarity?
The ...
- 5,630
12
votes
5
answers
5k
views
How to overcome training example's different lengths when working with Word Embeddings (word2vec)
I'm working on Sentiment Analysis over tweets using word2vec as word representation.
I have trained my word2vec model. But when I'm going to train my classifier, I'm facing the issue that every tweet ...
- 221
12
votes
2
answers
7k
views
Features of word vectors in Word2Vec
I am trying to do sentiment analysis. In order to convert the words to word vectors, I am using Word2Vec model. Suppose I have all the sentences in a list named 'sentences' and I am passing these ...
- 3,021
10
votes
2
answers
5k
views
NN embedding layer
Several neural network libraries such as tensorflow and pytorch offer an Embedding layer. Having implemented word2vec in the past, I understand the reasoning behind wanting a lower dimensional ...
- 101
10
votes
2
answers
11k
views
What is the difference between and Embedding Layer and an Autoencoder?
I'm reading about Embedding layers, especially applied to NLP and word2vec, and they seem nothing more than an application of Autoencoders for dimensionality reduction. Are they different? If so, what ...
- 6,065
10
votes
2
answers
12k
views
Reducing the dimensionality of word embeddings
I trained word embeddings with 300 dimensions. Now, I would like to have word embeddings with 50 dimensions: is it better to retrain the word embeddings with 50 dimensions, or can I use some ...
- 5,630
10
votes
3
answers
3k
views
Are Word2Vec and Doc2Vec both distributional representation or distributed representation?
I have read that distributional representation is based on distributional hypothesis that words occurring in similar context tends to have similar meanings.
Word2Vec and Doc2Vec both are modeled ...
- 1,954
10
votes
3
answers
3k
views
How the embedding layer is trained in Keras Embedding layer
How is the embedding layer trained in Keras Embedding layer?
(say using tensorflow backend, meaning is it similar to word2vec, glove or fasttext)
Assume we do not use a pretrained embedding.
- 755
10
votes
1
answer
2k
views
Text-Classification-Problem: Is Word2Vec/NN the best approach?
I am looking to design a system that given a paragraph of text will be able to categorize it and identify the context:
Is trained with user generated text paragraphs (like comments/questions/answers)
...
- 101
10
votes
1
answer
5k
views
How much training data does Word2Vec need?
I'd like to compare the difference among the same word mentioned in different sources. That is, how authors differ in their usage of ill-defined words, such as "democracy".
A brief plan was
Take the ...
- 711
9
votes
1
answer
7k
views
Why is the cosine distance used to measure the similatiry between word embeddings?
While computing the similarity between the words, cosine similarity or distance is computed on word vectors. Why aren't other distance metrics such as Euclidean ...
- 1,059
9
votes
1
answer
7k
views
Proper masking in the transformer model
Concerning the transformer model, a mask is used to mask out attention scores (replace with 1e-9) prior to the matrix multiplication with the value tensor. Regarding the masking, I have 3 short ...
- 193
8
votes
2
answers
16k
views
Ratio between embedded vector dimensions and vocabulary size
Using Embedding layer in Keras on a fairly small vocabulary (~300), I am looking at how to choose the output of this layer (dense vector) when given a 300 dimension ...
- 373
8
votes
1
answer
8k
views
K-means clustering of word embedding gives strange results
I'm trying to cluster words based on pre trained embeddings. I ran a simple experiment where I obtained around 100 words relating to "food taste", obtained word embeddings from a pre-trained set, and ...
- 111
8
votes
2
answers
7k
views
How should I use BERT embeddings for clustering (as opposed to fine-tuning BERT model for a supervised task)
First of all, I want to say that I am asking this question because I am interested in using BERT embeddings as document features to do clustering. I am using Transformers from the Hugging Face library....
- 805
8
votes
1
answer
4k
views
Difference between Gensim word2vec and keras Embedding layer
I used the gensim word2vec package and Keras Embedding layer for various different projects. Then I realize they seem to do the ...
- 2,725
8
votes
2
answers
6k
views
Text similarity with sentence embeddings
I'm trying to calculate similarity between texts with various lengths. My current approach is following:
Using Universal Sentence Encoder, I convert text to a set of vectors.
I average these vectors ...
8
votes
2
answers
2k
views
How does word2vec handle the input word being in the context?
If word2vec encounters the same word multiple times in the same window, what occurs? Obviously it is meaningless to decrease the distance between the vectors for the input word and the target word. ...
- 3,077
8
votes
2
answers
6k
views
How word2vec can handle unseen / new words to bypass this for new classifications?
In simple terms, if my classification is based on word2vec as features, what I am supposed to do, if a new word comes, which does not have a word2vec?
I am trying to used word2vec or word vectors for ...
- 81
8
votes
2
answers
1k
views
Averaging two Word2vec vectors to obtain a unified representation for single word
I have been working on a trained data for Word2vec algorithm. Since we need words to stay as original we don't make them lowercase at the preprocessing phase. Thus there are words with different ...
- 225
7
votes
1
answer
5k
views
Can we compare a word2vec vector with a doc2vec vector?
I have set of categories and I want to compare a document vector with word vector of categories to find best matching category.
Is it possible to compare a word vector with document vector? If yes, ...
- 3,775
7
votes
1
answer
9k
views
How to user Keras's Embedding Layer properly?
I'm a bit confused the proper usage of Embedding layer in Keras for seq2seq purpose (I'd like to reconstruct the TensorFlow se2seq machine translation tutorial in Keras). My questions are the ...
- 8,457
7
votes
3
answers
3k
views
meaning of fine-tuning in nlp task
There are two types of transfer learning model. One is
feature extraction, where the weights of the pre-trained model are not changed while training on the actual task and other is the weights of the ...
- 521
7
votes
1
answer
1k
views
what actually word embedding dimensions values represent?
I am learning word2vec and word embedding , I have downloaded GloVe pre-trained word embedding (shape 40,000 x 50) and using this function to extract information from that:
...
- 415
7
votes
1
answer
1k
views
How much text is enough to train a good embedding model?
I need to train a word2vec embedding model on Wikipedia articles using Gensim.
Eventually, I will use the entire Wikipedia for that
but for the moment, I'm doing some experimentation/optimization to ...
- 221
7
votes
3
answers
4k
views
Why is 10000 used as the denominator in Positional Encodings in the Transformer Model?
I was working through the All you need is Attention paper, and while the motivation of positional encodings makes sense and the other stackexchange answers filled me in on the motivations of the ...
7
votes
1
answer
9k
views
How is WordPiece tokenization helpful to effectively deal with rare words problem in NLP?
I have seen that NLP models such as BERT utilize WordPiece for tokenization. In WordPiece, we split the tokens like playing to play and ##ing. It is mentioned that it covers a wider spectrum of Out-Of-...
- 696
6
votes
2
answers
6k
views
What does dimension represent in GloVe pre-trained word vectors?
I'm using GloVe pre-trained word vectors (glove.6b.50d.txt, glove.6b.300d.txt) as word embedding.
I have a conceptual question:
...
- 163
6
votes
2
answers
8k
views
What is the vector value of [CLS] [SEP] tokens in BERT
In BERT, They replace separator and start of sentence with special token labels. What are there corresponding values in embedding_matrix. Are they 0-vector?
I wanted to replace the proper nouns like ...
- 251
6
votes
1
answer
2k
views
Sum vs mean of word-embeddings for sentence similarity
So, say I have the following sentences
["The dog says woof", "a king leads the country", "an apple is red"]
I can embed each word using an ...
- 460
6
votes
1
answer
1k
views
What are the advantages of Skip-Gram methods?
Concerning the notion of word embeddings, Skip-Gram methods aim for computing the probability of a word given its neighborhood. I do not understand the rationale behind it, since it is possible to ...
- 241
6
votes
3
answers
7k
views
Using several documents with word2vec
I want to train word embeddings using word2vec. My corpus is split into several documents (it's a large set of patient notes). Should I just concatenate all documents into one before running word2vec ...
- 5,630
6
votes
2
answers
1k
views
Word2Vec: Why do some dimensions of an embedding have an interpretation, and why does addition/subtraction of embedding vectors work?
I'm reading about Word2Vec from this source: http://jalammar.github.io/illustrated-word2vec/. Below is the heatmap of the embeddings for various words. In the source, it's claimed that we can get an ...
- 299
6
votes
1
answer
9k
views
What GPU size do I need to fine tune BERT base cased?
I want to fine tune BERT Multilingual but I'm not aware about the GPU requirements to train BERT Multilingual. I have GTX 1050ti 4GB on my local machine. I want to know what size of GPU is needed and ...
6
votes
1
answer
6k
views
Using Trainable=True in Keras Embedding obtained better performance
It is suggested by the author of Keras [1] to use Trainable=False when using the embedding layer in Keras to prevent the weights from being updated during training. ...
- 163
6
votes
1
answer
858
views
What is the neural network architecture behind Facebook's Starspace model?
Recently, Facebook released a paper concerning a general purpose neural embedding model called StarSpace.
In their paper, they explain the loss function and the training procedure of the model, but ...
- 547
6
votes
2
answers
4k
views
How to handle Memory issues in training Word Embeddings on Large Datasets?
I want to train a word predictability task to generate word embeddings. The document collection contains 243k documents. The code implementation is in torch. I am struggling with the huge size of the ...
- 163
5
votes
1
answer
11k
views
How pre-trained BERT model generates word embeddings for out of vocabulary words?
Currently, I am reading BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. I want to understand how pre-trained BERT generates word embeddings for out of vocabulary ...
- 2,041
5
votes
3
answers
3k
views
Why using a frozen embedding layer in an LSTM model
I'm studying this LSTM mode:
https://www.kaggle.com/paoloripamonti/twitter-sentiment-analysis
They use a frozen embedding layer which uses an predefined matrix with for each word a 300 dim vector ...
- 153
5
votes
2
answers
1k
views
Why is averaging the vectors required in word2vec?
While implementing word2vec using gensim by following few tutorials online, one thing that I couldn't understand is the reason why word vectors are averaged once the model is trained. Few example ...
- 163