Skip to main content
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more

Questions tagged [word-embeddings]

Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.

Filter by
Sorted by
Tagged with
5 votes
1 answer
12k views

How pre-trained BERT model generates word embeddings for out of vocabulary words?

Currently, I am reading BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. I want to understand how pre-trained BERT generates word embeddings for out of vocabulary ...
Sayali Sonawane's user avatar
22 votes
2 answers
18k views

Doc2Vec - How to label the paragraphs (gensim)

I am wondering how to label (tag) sentences / paragraphs / documents with doc2vec in gensim - from a practical standpoint. Do you need to have each sentence / paragraph / document with its own ...
B_Miner's user avatar
  • 702
13 votes
3 answers
5k views

Why do we need 2 matrices for word2vec or GloVe

Word2vec and GloVe are the two most known words embedding methods. Many works pointed that these two models are actually very close to each other and that under some assumptions, they perform a matrix ...
Robin's user avatar
  • 1,337
28 votes
2 answers
24k views

Predicting a word using Word2vec model

Given a sentence: "When I open the ?? door it starts heating automatically" I would like to get the list of possible words in ?? with a probability. The basic concept used in word2vec model ...
DED's user avatar
  • 345
6 votes
2 answers
8k views

What is the vector value of [CLS] [SEP] tokens in BERT

In BERT, They replace separator and start of sentence with special token labels. What are there corresponding values in embedding_matrix. Are they 0-vector? I wanted to replace the proper nouns like ...
Itachi's user avatar
  • 251
1 vote
2 answers
982 views

How to obtain vector representation of phrases using the embedding layer and do PCA with it

I am trying to understand from both a conceptual and a Python code point of view, how to represent phrases that are present in a corpus (that is used to train a neural network to classify phrases) as ...
HelpNeederStudent's user avatar
0 votes
2 answers
5k views

How to JUST represent words as embeddings by pretrained BERT?

I don't have enough data (i.e. I don't have enough texts) --- have only around 4k words in my dictionary. I need to compare given words, then I need to representate it as embedding. After the ...
taciturno's user avatar
  • 137
44 votes
5 answers
43k views

Intuitive explanation of Noise Contrastive Estimation (NCE) loss?

I read about NCE (a form of candidate sampling) from these two sources: Tensorflow writeup Original Paper Can someone help me with the following: A simple explanation of how NCE works (I found the ...
tejaskhot's user avatar
  • 4,075
33 votes
5 answers
40k views

How can I get a measure of the semantic similarity of words?

What is the best way to figure out the semantic similarity of words? Word2Vec is okay, but not ideal: ...
Thomas Johnson's user avatar
10 votes
3 answers
3k views

How the embedding layer is trained in Keras Embedding layer

How is the embedding layer trained in Keras Embedding layer? (say using tensorflow backend, meaning is it similar to word2vec, glove or fasttext) Assume we do not use a pretrained embedding.
william007's user avatar
10 votes
1 answer
5k views

How much training data does Word2Vec need?

I'd like to compare the difference among the same word mentioned in different sources. That is, how authors differ in their usage of ill-defined words, such as "democracy". A brief plan was Take the ...
Anton Tarasenko's user avatar
8 votes
1 answer
4k views

Difference between Gensim word2vec and keras Embedding layer

I used the gensim word2vec package and Keras Embedding layer for various different projects. Then I realize they seem to do the ...
Edamame's user avatar
  • 2,755
8 votes
2 answers
2k views

Averaging two Word2vec vectors to obtain a unified representation for single word

I have been working on a trained data for Word2vec algorithm. Since we need words to stay as original we don't make them lowercase at the preprocessing phase. Thus there are words with different ...
ozgur's user avatar
  • 225
6 votes
1 answer
10k views

What GPU size do I need to fine tune BERT base cased?

I want to fine tune BERT Multilingual but I'm not aware about the GPU requirements to train BERT Multilingual. I have GTX 1050ti 4GB on my local machine. I want to know what size of GPU is needed and ...
Darshan Bhandari's user avatar
5 votes
3 answers
7k views

where to store embeddings for similarity search?

I've asked on stackoverflow already (here), but I figured that the approach of storing embeddings in an ordinary postgres-Database might be flawed from the very beginning. I will shortly etch out the ...
Angus's user avatar
  • 51
4 votes
3 answers
6k views

Word embedding of a new word which was not in training

Let's say I trained a Skip-Gram model (Word2Vec) for my vocabulary of size 10,000. The representation allows me to reduce the dimension from 10,000 (one-hot-encoding) to 100 (size of hidden layer of ...
Victor's user avatar
  • 281
4 votes
2 answers
4k views

doc2vec - How does the inference step work in PV-DBOW

I am quite confused about how we generate new paragraph vectors in PV-DBOW? If I want to use the embeddings to classify some text how would I generate a vector for a new paragraph? In the original ...
Kevin's user avatar
  • 261
4 votes
2 answers
3k views

Document similarity: Vector embedding versus BoW performance?

I have a collection of documents, where each document is rapidly growing with time. The task is to find similar documents at any fixed time. I have two potential approaches: A vector embedding (...
Alec Matusis's user avatar
3 votes
3 answers
7k views

What is word embedding and character embedding ? Why words are represented in vector with huge size?

In NLP word embedding represent word as number but after reading many blog i found that word are represent as vectors ? so what is word embedding exactly and Why words are represented in vector and ...
star's user avatar
  • 1,471
3 votes
2 answers
3k views

When to use different Word2Vec training approaches?

So I am learning Word2Vec for the first time and my question is quite basic: How to know what approach to use? I.e, Word2Vec in Tensorflow or Word2Vec trained with Gensim ? In what cases would ...
ocean800's user avatar
  • 147
2 votes
1 answer
4k views

Is it possible to add new vocabulary to BERT's tokenizer when fine-tuning?

I want to fine-tune BERT by training it on a domain dataset of my own. The domain is specific and includes many terms that probably weren't included in the original dataset BERT was trained on. I know ...
user123635's user avatar
2 votes
1 answer
4k views

What are the approaches to aggregate categorical variables?

I am working on a clickstream dataset. I have come up with the following example dataset to explain my problem: ...
Amir's user avatar
  • 123
2 votes
0 answers
203 views

NER at sentence level or document level?

Should NER models (LSTM or CRF) take input training data at sentence level or paragraph level? Let's say we have this input text, and we would like to do Named Entity Extraction: GOP Sen. Rand ...
Franklin Dong's user avatar
1 vote
1 answer
173 views

Why the label is not explicitly involved in the loss function of skip-gram?

I am recently learning word embedding myself. When learning skip-gram from the paper https://arxiv.org/pdf/1310.4546.pdf[Distributed Representations of Words and Phrases and their Compositionality], I ...
JQ_SE's user avatar
  • 11
1 vote
1 answer
2k views

How can i get the vector of word using BERT?

I need to get word-vectors using BERT and got this function that i think it should be the one i need ...
user5520049's user avatar
1 vote
2 answers
3k views

Dealing with multiple distinct-value categorical variables

So, I've got a dataset with almost all of its columns are categorical variables. Problem is that most of the categorical variables have so many distinct values. For instance, one column have more ...
Abdullah Mohamed's user avatar
0 votes
1 answer
693 views

Where the objective function of Skip-Gram Negative Sampling (SGNS) come from?

In the paper of "Neural Word Embedding as Implicit Matrix Factorization", there is an objective function for Skip-Gram Negative Sampling. I wanted to know where this formula come from
Mahdi Amrollahi's user avatar