Questions tagged [word-embeddings]

Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.

Filter by
Sorted by
Tagged with
44 votes
5 answers
42k views

Intuitive explanation of Noise Contrastive Estimation (NCE) loss?

I read about NCE (a form of candidate sampling) from these two sources: Tensorflow writeup Original Paper Can someone help me with the following: A simple explanation of how NCE works (I found the ...
tejaskhot's user avatar
  • 4,035
33 votes
5 answers
40k views

How can I get a measure of the semantic similarity of words?

What is the best way to figure out the semantic similarity of words? Word2Vec is okay, but not ideal: ...
Thomas Johnson's user avatar
30 votes
3 answers
20k views

What is a better input for Word2Vec?

This is more like a general NLP question. What is the appropriate input to train a word embedding namely Word2Vec? Should all sentences belonging to an article be a separate document in a corpus? Or ...
wacax's user avatar
  • 3,390
28 votes
2 answers
24k views

Predicting a word using Word2vec model

Given a sentence: "When I open the ?? door it starts heating automatically" I would like to get the list of possible words in ?? with a probability. The basic concept used in word2vec model ...
DED's user avatar
  • 345
27 votes
5 answers
20k views

BERT vs Word2VEC: Is bert disambiguating the meaning of the word vector?

Word2vec: Word2vec provides a vector for each token/word and those vectors encode the meaning of the word. Although those vectors are not human interpretable, the meaning of the vectors are ...
sovon's user avatar
  • 521
22 votes
2 answers
18k views

Doc2Vec - How to label the paragraphs (gensim)

I am wondering how to label (tag) sentences / paragraphs / documents with doc2vec in gensim - from a practical standpoint. Do you need to have each sentence / paragraph / document with its own ...
B_Miner's user avatar
  • 702
21 votes
4 answers
44k views

How to initialize a new word2vec model with pre-trained model weights?

I am using Gensim Library in python for using and training word2vector model. Recently, I was looking at initializing my model weights with some pre-trained word2vec model such as (GoogleNewDataset ...
Nomiluks's user avatar
  • 461
16 votes
2 answers
31k views

One Hot Encoding vs Word Embedding - When to choose one or another?

A colleague of mine is having an interesting situation, he has quite a large set of possibilities for a defined categorical feature (+/- 300 different values) The usual data science approach would be ...
Jonathan DEKHTIAR's user avatar
14 votes
4 answers
18k views

How word2vec can be used to identify unseen words and relate them to already trained data

I was working on word2vec gensim model and found it really interesting. I am intersted in finding how a unknown/unseen word when checked with the model will be able to get similar terms from the ...
gaurus's user avatar
  • 341
13 votes
3 answers
5k views

Why do we need 2 matrices for word2vec or GloVe

Word2vec and GloVe are the two most known words embedding methods. Many works pointed that these two models are actually very close to each other and that under some assumptions, they perform a matrix ...
Robin's user avatar
  • 1,337
12 votes
2 answers
10k views

Shall I use the Euclidean Distance or the Cosine Similarity to compute the semantic similarity of two words?

I want to compute the semantic similarity of two words using their vector representations (obtained using e.g. word2vec, GloVe, etc.). Shall I use the Euclidean Distance or the Cosine Similarity? The ...
Franck Dernoncourt's user avatar
12 votes
5 answers
6k views

How to overcome training example's different lengths when working with Word Embeddings (word2vec)

I'm working on Sentiment Analysis over tweets using word2vec as word representation. I have trained my word2vec model. But when I'm going to train my classifier, I'm facing the issue that every tweet ...
antorqs's user avatar
  • 221
12 votes
2 answers
8k views

Features of word vectors in Word2Vec

I am trying to do sentiment analysis. In order to convert the words to word vectors, I am using Word2Vec model. Suppose I have all the sentences in a list named 'sentences' and I am passing these ...
enterML's user avatar
  • 3,031
11 votes
2 answers
11k views

What is the difference between and Embedding Layer and an Autoencoder?

I'm reading about Embedding layers, especially applied to NLP and word2vec, and they seem nothing more than an application of Autoencoders for dimensionality reduction. Are they different? If so, what ...
Leevo's user avatar
  • 6,175
10 votes
2 answers
5k views

NN embedding layer

Several neural network libraries such as tensorflow and pytorch offer an Embedding layer. Having implemented word2vec in the past, I understand the reasoning behind wanting a lower dimensional ...
cbake's user avatar
  • 101
10 votes
2 answers
12k views

Reducing the dimensionality of word embeddings

I trained word embeddings with 300 dimensions. Now, I would like to have word embeddings with 50 dimensions: is it better to retrain the word embeddings with 50 dimensions, or can I use some ...
Franck Dernoncourt's user avatar
10 votes
3 answers
3k views

Are Word2Vec and Doc2Vec both distributional representation or distributed representation?

I have read that distributional representation is based on distributional hypothesis that words occurring in similar context tends to have similar meanings. Word2Vec and Doc2Vec both are modeled ...
chmodsss's user avatar
  • 1,964
10 votes
3 answers
3k views

How the embedding layer is trained in Keras Embedding layer

How is the embedding layer trained in Keras Embedding layer? (say using tensorflow backend, meaning is it similar to word2vec, glove or fasttext) Assume we do not use a pretrained embedding.
william007's user avatar
10 votes
1 answer
2k views

Text-Classification-Problem: Is Word2Vec/NN the best approach?

I am looking to design a system that given a paragraph of text will be able to categorize it and identify the context: Is trained with user generated text paragraphs (like comments/questions/answers) ...
Shankar's user avatar
  • 101
10 votes
1 answer
5k views

How much training data does Word2Vec need?

I'd like to compare the difference among the same word mentioned in different sources. That is, how authors differ in their usage of ill-defined words, such as "democracy". A brief plan was Take the ...
Anton Tarasenko's user avatar
9 votes
1 answer
7k views

Why is the cosine distance used to measure the similatiry between word embeddings?

While computing the similarity between the words, cosine similarity or distance is computed on word vectors. Why aren't other distance metrics such as Euclidean ...
Ashwin Geet D'Sa's user avatar
9 votes
1 answer
7k views

Proper masking in the transformer model

Concerning the transformer model, a mask is used to mask out attention scores (replace with 1e-9) prior to the matrix multiplication with the value tensor. Regarding the masking, I have 3 short ...
beginneR's user avatar
  • 193
8 votes
2 answers
16k views

Ratio between embedded vector dimensions and vocabulary size

Using Embedding layer in Keras on a fairly small vocabulary (~300), I am looking at how to choose the output of this layer (dense vector) when given a 300 dimension ...
0xmax's user avatar
  • 373
8 votes
1 answer
8k views

K-means clustering of word embedding gives strange results

I'm trying to cluster words based on pre trained embeddings. I ran a simple experiment where I obtained around 100 words relating to "food taste", obtained word embeddings from a pre-trained set, and ...
Thusitha's user avatar
  • 111
8 votes
2 answers
7k views

How should I use BERT embeddings for clustering (as opposed to fine-tuning BERT model for a supervised task)

First of all, I want to say that I am asking this question because I am interested in using BERT embeddings as document features to do clustering. I am using Transformers from the Hugging Face library....
fractalnature's user avatar
8 votes
1 answer
4k views

Difference between Gensim word2vec and keras Embedding layer

I used the gensim word2vec package and Keras Embedding layer for various different projects. Then I realize they seem to do the ...
Edamame's user avatar
  • 2,745
8 votes
2 answers
6k views

Text similarity with sentence embeddings

I'm trying to calculate similarity between texts with various lengths. My current approach is following: Using Universal Sentence Encoder, I convert text to a set of vectors. I average these vectors ...
Kertis van Kertis's user avatar
8 votes
2 answers
2k views

How does word2vec handle the input word being in the context?

If word2vec encounters the same word multiple times in the same window, what occurs? Obviously it is meaningless to decrease the distance between the vectors for the input word and the target word. ...
jamesmf's user avatar
  • 3,087
8 votes
2 answers
6k views

How word2vec can handle unseen / new words to bypass this for new classifications?

In simple terms, if my classification is based on word2vec as features, what I am supposed to do, if a new word comes, which does not have a word2vec? I am trying to used word2vec or word vectors for ...
Sarath's user avatar
  • 81
8 votes
3 answers
4k views

Why is 10000 used as the denominator in Positional Encodings in the Transformer Model?

I was working through the All you need is Attention paper, and while the motivation of positional encodings makes sense and the other stackexchange answers filled me in on the motivations of the ...
ThirtyOneTwentySeven's user avatar
8 votes
2 answers
2k views

Averaging two Word2vec vectors to obtain a unified representation for single word

I have been working on a trained data for Word2vec algorithm. Since we need words to stay as original we don't make them lowercase at the preprocessing phase. Thus there are words with different ...
ozgur's user avatar
  • 225
7 votes
1 answer
5k views

Can we compare a word2vec vector with a doc2vec vector?

I have set of categories and I want to compare a document vector with word vector of categories to find best matching category. Is it possible to compare a word vector with document vector? If yes, ...
SHASHANK GUPTA's user avatar
7 votes
1 answer
2k views

Sum vs mean of word-embeddings for sentence similarity

So, say I have the following sentences ["The dog says woof", "a king leads the country", "an apple is red"] I can embed each word using an ...
CutePoison's user avatar
7 votes
1 answer
9k views

How to user Keras's Embedding Layer properly?

I'm a bit confused the proper usage of Embedding layer in Keras for seq2seq purpose (I'd like to reconstruct the TensorFlow se2seq machine translation tutorial in Keras). My questions are the ...
Hendrik's user avatar
  • 8,557
7 votes
3 answers
3k views

meaning of fine-tuning in nlp task

There are two types of transfer learning model. One is feature extraction, where the weights of the pre-trained model are not changed while training on the actual task and other is the weights of the ...
sovon's user avatar
  • 521
7 votes
1 answer
1k views

what actually word embedding dimensions values represent?

I am learning word2vec and word embedding , I have downloaded GloVe pre-trained word embedding (shape 40,000 x 50) and using this function to extract information from that: ...
Aaditya ura's user avatar
7 votes
1 answer
1k views

How much text is enough to train a good embedding model?

I need to train a word2vec embedding model on Wikipedia articles using Gensim. Eventually, I will use the entire Wikipedia for that but for the moment, I'm doing some experimentation/optimization to ...
Abdulrahman Bres's user avatar
7 votes
1 answer
9k views

How is WordPiece tokenization helpful to effectively deal with rare words problem in NLP?

I have seen that NLP models such as BERT utilize WordPiece for tokenization. In WordPiece, we split the tokens like playing to play and ##ing. It is mentioned that it covers a wider spectrum of Out-Of-...
Harman's user avatar
  • 706
6 votes
2 answers
6k views

What does dimension represent in GloVe pre-trained word vectors?

I'm using GloVe pre-trained word vectors (glove.6b.50d.txt, glove.6b.300d.txt) as word embedding. I have a conceptual question: ...
Benyamin Jafari's user avatar
6 votes
2 answers
8k views

What is the vector value of [CLS] [SEP] tokens in BERT

In BERT, They replace separator and start of sentence with special token labels. What are there corresponding values in embedding_matrix. Are they 0-vector? I wanted to replace the proper nouns like ...
Itachi's user avatar
  • 251
6 votes
1 answer
1k views

What are the advantages of Skip-Gram methods?

Concerning the notion of word embeddings, Skip-Gram methods aim for computing the probability of a word given its neighborhood. I do not understand the rationale behind it, since it is possible to ...
Jorgemar's user avatar
  • 241
6 votes
3 answers
7k views

Using several documents with word2vec

I want to train word embeddings using word2vec. My corpus is split into several documents (it's a large set of patient notes). Should I just concatenate all documents into one before running word2vec ...
Franck Dernoncourt's user avatar
6 votes
2 answers
2k views

Word2Vec: Why do some dimensions of an embedding have an interpretation, and why does addition/subtraction of embedding vectors work?

I'm reading about Word2Vec from this source: http://jalammar.github.io/illustrated-word2vec/. Below is the heatmap of the embeddings for various words. In the source, it's claimed that we can get an ...
Shirish Kulhari's user avatar
6 votes
1 answer
10k views

What GPU size do I need to fine tune BERT base cased?

I want to fine tune BERT Multilingual but I'm not aware about the GPU requirements to train BERT Multilingual. I have GTX 1050ti 4GB on my local machine. I want to know what size of GPU is needed and ...
Darshan Bhandari's user avatar
6 votes
1 answer
6k views

Using Trainable=True in Keras Embedding obtained better performance

It is suggested by the author of Keras [1] to use Trainable=False when using the embedding layer in Keras to prevent the weights from being updated during training. ...
sugab's user avatar
  • 163
6 votes
1 answer
860 views

What is the neural network architecture behind Facebook's Starspace model?

Recently, Facebook released a paper concerning a general purpose neural embedding model called StarSpace. In their paper, they explain the loss function and the training procedure of the model, but ...
ChiPlusPlus's user avatar
6 votes
3 answers
2k views

Best way to vectorise names and addresses for similarity searching?

I have a large dataset of around 9 million people with names and addresses. Given quirks of the process used to get the data it is highly likely that a person is in the dataset more than once, with ...
Sandy Lee's user avatar
  • 247
6 votes
2 answers
4k views

How to handle Memory issues in training Word Embeddings on Large Datasets?

I want to train a word predictability task to generate word embeddings. The document collection contains 243k documents. The code implementation is in torch. I am struggling with the huge size of the ...
Kahini Wadhawan's user avatar
5 votes
1 answer
12k views

How pre-trained BERT model generates word embeddings for out of vocabulary words?

Currently, I am reading BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. I want to understand how pre-trained BERT generates word embeddings for out of vocabulary ...
Sayali Sonawane's user avatar
5 votes
3 answers
3k views

Why using a frozen embedding layer in an LSTM model

I'm studying this LSTM mode: https://www.kaggle.com/paoloripamonti/twitter-sentiment-analysis They use a frozen embedding layer which uses an predefined matrix with for each word a 300 dim vector ...
sg_sg94's user avatar
  • 153

1
2 3 4 5
11