Questions tagged [word-embeddings]
Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.
26
questions
5
votes
1
answer
11k
views
How pre-trained BERT model generates word embeddings for out of vocabulary words?
Currently, I am reading BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. I want to understand how pre-trained BERT generates word embeddings for out of vocabulary ...
- 2,041
22
votes
2
answers
18k
views
Doc2Vec - How to label the paragraphs (gensim)
I am wondering how to label (tag) sentences / paragraphs / documents with doc2vec in gensim - from a practical standpoint.
Do you need to have each sentence / paragraph / document with its own ...
- 702
13
votes
3
answers
5k
views
Why do we need 2 matrices for word2vec or GloVe
Word2vec and GloVe are the two most known words embedding methods. Many works pointed that these two models are actually very close to each other and that under some assumptions, they perform a matrix ...
- 1,307
28
votes
2
answers
24k
views
Predicting a word using Word2vec model
Given a sentence:
"When I open the ?? door it starts heating automatically"
I would like to get the list of possible words in ?? with a probability.
The basic concept used in word2vec model ...
- 345
6
votes
2
answers
8k
views
What is the vector value of [CLS] [SEP] tokens in BERT
In BERT, They replace separator and start of sentence with special token labels. What are there corresponding values in embedding_matrix. Are they 0-vector?
I wanted to replace the proper nouns like ...
- 251
1
vote
2
answers
883
views
How to obtain vector representation of phrases using the embedding layer and do PCA with it
I am trying to understand from both a conceptual and a Python code point of view, how to represent phrases that are present in a corpus (that is used to train a neural network to classify phrases) as ...
0
votes
2
answers
4k
views
How to JUST represent words as embeddings by pretrained BERT?
I don't have enough data (i.e. I don't have enough texts) --- have only around 4k words in my dictionary. I need to compare given words, then I need to representate it as embedding.
After the ...
- 137
44
votes
5
answers
41k
views
Intuitive explanation of Noise Contrastive Estimation (NCE) loss?
I read about NCE (a form of candidate sampling) from these two sources:
Tensorflow writeup
Original Paper
Can someone help me with the following:
A simple explanation of how NCE works (I found the ...
- 4,025
33
votes
5
answers
40k
views
How can I get a measure of the semantic similarity of words?
What is the best way to figure out the semantic similarity of words? Word2Vec is okay, but not ideal:
...
- 665
10
votes
1
answer
5k
views
How much training data does Word2Vec need?
I'd like to compare the difference among the same word mentioned in different sources. That is, how authors differ in their usage of ill-defined words, such as "democracy".
A brief plan was
Take the ...
- 711
10
votes
3
answers
3k
views
How the embedding layer is trained in Keras Embedding layer
How is the embedding layer trained in Keras Embedding layer?
(say using tensorflow backend, meaning is it similar to word2vec, glove or fasttext)
Assume we do not use a pretrained embedding.
- 755
8
votes
2
answers
1k
views
Averaging two Word2vec vectors to obtain a unified representation for single word
I have been working on a trained data for Word2vec algorithm. Since we need words to stay as original we don't make them lowercase at the preprocessing phase. Thus there are words with different ...
- 225
8
votes
1
answer
4k
views
Difference between Gensim word2vec and keras Embedding layer
I used the gensim word2vec package and Keras Embedding layer for various different projects. Then I realize they seem to do the ...
- 2,735
6
votes
1
answer
9k
views
What GPU size do I need to fine tune BERT base cased?
I want to fine tune BERT Multilingual but I'm not aware about the GPU requirements to train BERT Multilingual. I have GTX 1050ti 4GB on my local machine. I want to know what size of GPU is needed and ...
5
votes
3
answers
7k
views
where to store embeddings for similarity search?
I've asked on stackoverflow already (here), but I figured that the approach of storing embeddings in an ordinary postgres-Database might be flawed from the very beginning. I will shortly etch out the ...
- 51
4
votes
2
answers
4k
views
doc2vec - How does the inference step work in PV-DBOW
I am quite confused about how we generate new paragraph vectors in PV-DBOW?
If I want to use the embeddings to classify some text how would I generate a vector for a new paragraph?
In the original ...
- 261
4
votes
2
answers
3k
views
Document similarity: Vector embedding versus BoW performance?
I have a collection of documents, where each document is rapidly growing with time. The task is to find similar documents at any fixed time. I have two potential approaches:
A vector embedding (...
- 149
4
votes
3
answers
6k
views
Word embedding of a new word which was not in training
Let's say I trained a Skip-Gram model (Word2Vec) for my vocabulary of size 10,000.
The representation allows me to reduce the dimension from 10,000 (one-hot-encoding) to 100 (size of hidden layer of ...
- 271
3
votes
3
answers
6k
views
What is word embedding and character embedding ? Why words are represented in vector with huge size?
In NLP word embedding represent word as number but after reading many blog i found that word are represent as vectors ? so what is word embedding exactly and Why words are represented in vector and ...
- 1,431
3
votes
2
answers
3k
views
When to use different Word2Vec training approaches?
So I am learning Word2Vec for the first time and my question is quite basic: How to know what approach to use? I.e, Word2Vec in Tensorflow or Word2Vec trained with Gensim ?
In what cases would ...
- 147
2
votes
0
answers
193
views
NER at sentence level or document level?
Should NER models (LSTM or CRF) take input training data at sentence level or paragraph level?
Let's say we have this input text, and we would like to do Named Entity Extraction:
GOP Sen. Rand ...
2
votes
1
answer
3k
views
Is it possible to add new vocabulary to BERT's tokenizer when fine-tuning?
I want to fine-tune BERT by training it on a domain dataset of my own. The domain is specific and includes many terms that probably weren't included in the original dataset BERT was trained on. I know ...
- 21
2
votes
1
answer
4k
views
What are the approaches to aggregate categorical variables?
I am working on a clickstream dataset. I have come up with the following example dataset to explain my problem:
...
- 123
1
vote
1
answer
2k
views
How can i get the vector of word using BERT?
I need to get word-vectors using BERT and got this function that i think it should be the one i need
...
- 113
1
vote
2
answers
3k
views
Dealing with multiple distinct-value categorical variables
So, I've got a dataset with almost all of its columns are categorical variables. Problem is that most of the categorical variables have so many distinct values.
For instance, one column have more ...
0
votes
1
answer
551
views
Where the objective function of Skip-Gram Negative Sampling (SGNS) come from?
In the paper of "Neural Word Embedding as Implicit Matrix Factorization", there is an objective function for Skip-Gram Negative Sampling. I wanted to know where this formula come from
- 263