Questions tagged [word-embeddings]
Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.
507
questions
5
votes
1
answer
11k
views
How pre-trained BERT model generates word embeddings for out of vocabulary words?
Currently, I am reading BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. I want to understand how pre-trained BERT generates word embeddings for out of vocabulary ...
- 2,041
2
votes
0
answers
275
views
How to use paraphrase_mining using sentence transformers pre-trained model
I am trying to find similarity between sentences using a pre-trained sentence-transformers model. I am trying to follow the code here - https://www.sbert.net/docs/usage/paraphrase_mining.html
In trial ...
- 121
4
votes
1
answer
279
views
What is the structure and dimension of input passed to neural network when training CBOW and SKIP GRAM word embedding
I am confused about input passed to neural network in natural language processing (NLP) when training CBOW word embedding from scratch. I read the paper and have ...
- 277
2
votes
1
answer
303
views
DBSCAN on textual and numerical columns
I have a dataset which has two columns:
title price
sentence1 12
sentence2 13
I have used doc2vec to convert the ...
- 420
2
votes
1
answer
32
views
Does the output of the Sequence-to-Sequence encoder model exist in the same semantic space as the inputs (Word2vec)? [closed]
Does the output generated from the LSTM encoder module exist in the same semantic space as the original word vectors?
If so, say for example we have a sentence and we pass it through the encoder to ...
- 23
3
votes
1
answer
5k
views
How to choose dimension of Keras embedding layer?
Looking for some guidelines to choose dimension of Keras word embedding layer. For example in a simplified movie review classification code:
...
- 295
1
vote
0
answers
93
views
node2vec initial embedding
I am referring to the following implementation of node2vec: https://github.com/eliorc/node2vec
Is there a way to have a "smart initialization" with node2vec, i.e., to start the algorithm ...
- 31
1
vote
2
answers
529
views
Difference between CBOW and SKIP Gram word vectors
I have gone through several links but was not able to understand how CBOW and Skip Gram is trained from scratch?
Any good link/blogs or books would be very helpful.
...
- 277
2
votes
1
answer
174
views
Can we combine multiple K-Means Models as a single model?
I have a NLP problem statement where I use a Word2Vec embedding pre-trained model to convert key text to vectors and then on a set of terms run k-means clustering to get a final model for certain <...
- 125
7
votes
3
answers
4k
views
Why is 10000 used as the denominator in Positional Encodings in the Transformer Model?
I was working through the All you need is Attention paper, and while the motivation of positional encodings makes sense and the other stackexchange answers filled me in on the motivations of the ...
1
vote
1
answer
73
views
Initializing weights that are a pointwise product of multiple variables
In two-layer perceptrons that slide across words of text, such as word2vec and fastText, hidden layer heights may be a product of two random variables such as positional embeddings and word embeddings ...
- 111
1
vote
0
answers
157
views
Embed Sequence API in Tensorflow
My current tensorflow version is 2.1 and I'm using low level(core) tensorflow api's.
Please provide a substitute of "tf.contrib.layers.embed_sequence".
I have explored a lot, but could not ...
- 1,252
1
vote
0
answers
41
views
Learning word embeddings by first learning character embeddings
I was going through various papers for NLU applications(Natural Language Understanding).
There I have observed a common pattern that for a word embeddings, following 3 combinations are used (may be ...
- 11
2
votes
1
answer
146
views
What's the best way to detect bible verse mentions in a text?
I have a set of 10 verses from the Bible in English. I want to detect the occurrence of any of these verses in a text. What would be the best way to go about doing this?
Note that verses of the Bible ...
1
vote
1
answer
194
views
Construct word2vec (CBOW) training data from beginning of sentence
When constructing training data for CBOW, Mikolov et al. suggest using the word from the center of a context window. What is the "best" approach to capturing words at the beginning/end of a ...
- 21
9
votes
1
answer
7k
views
Why is the cosine distance used to measure the similatiry between word embeddings?
While computing the similarity between the words, cosine similarity or distance is computed on word vectors. Why aren't other distance metrics such as Euclidean ...
- 1,059
6
votes
1
answer
9k
views
What GPU size do I need to fine tune BERT base cased?
I want to fine tune BERT Multilingual but I'm not aware about the GPU requirements to train BERT Multilingual. I have GTX 1050ti 4GB on my local machine. I want to know what size of GPU is needed and ...
1
vote
1
answer
835
views
Can we use sentence transformers to embed sentences without labels?
I was trying to use this project :
https://github.com/UKPLab/sentence-transformers
for embedding non english sentences, the language is not a human speaking language, its machine language (x86)
but ...
- 191
8
votes
2
answers
7k
views
How should I use BERT embeddings for clustering (as opposed to fine-tuning BERT model for a supervised task)
First of all, I want to say that I am asking this question because I am interested in using BERT embeddings as document features to do clustering. I am using Transformers from the Hugging Face library....
- 805
0
votes
1
answer
101
views
Is adding the embedded words of a sentence to represent the sentence a good approach?
I have a dataset of sentences in a non english language like :
word1 word2 word3 word62
word5 word1 word2
Now i want to turn each variable length sentence to a fixed size vector to give it to my ...
- 191
0
votes
1
answer
477
views
best approach to embed random length sequences of words as a fixed size vector without having a maximum length? [closed]
I have a dataset of sentences in a non-English language like:
word1 word2 word3 word62
word5 word1 word2
and the length of each sentence is not fixed.
Now, I want to represent each sentence as a ...
- 191
3
votes
2
answers
2k
views
Should I keep common stop-words when preprocessing for word embedding?
If I want to construct a word embedding by predicting a target word given context words, is it better to remove stop words or keep them?
the quick brown fox jumped over the lazy dog
or
quick brown ...
- 145
1
vote
1
answer
37
views
How keras.layers.embedding learn word embeddings?
I was trying some tensorflow tutorials and see that in all of them they use layers.embedding to learn these word embeddings, but how are these learned? , with a NN? which arquitecture? , or word2vec?
...
- 33
0
votes
1
answer
252
views
how to train custom word2vec embeddings to find related articles?
I am beginner in machine learning. My project is to make search engine based on AI which shows related articles when we search on website. For this i decided to train my own embedding.
I found two ...
- 1
4
votes
2
answers
8k
views
Can we use BERT for only word embedding and then use SVM/RNN to do intent classification?
According to this article, "Systems used for intent classification contain the following two components:
Word embedding, and a classifier." This article also evaluated BERT+SVM and Word2Vec+...
- 41
1
vote
1
answer
394
views
Understanding Transfer Learning of Word Embeddings
I can't quite visualize how transfer learning of pre-trained word embeddings is useful in an NLP task( say named entity recognition ) . I'm studying Andrew NG's Sequence Models course and he seems to ...
- 225
4
votes
4
answers
860
views
word2vec: usefulness of context vectors in classification
I've been working on a NN-based classification system that accepts document vectors as input. I can't really talk about what I'm specifically training the neural net on, so i'm hoping for a more ...
- 268
2
votes
1
answer
373
views
How do I read the cord_19_embeddings_2020-07-16.csv from the COVID-19 Open Research Dataset Challenge (CORD-19) on Kaggle?
What I'm trying to do
I wanted to use the CORD19 word embeddings csv to map them to certain findings from the rest of the dataset, but as we can see there are no stings in the first column.
The way I ...
- 151
1
vote
2
answers
1k
views
How best to embed large and noisy documents
I have a large corpus of documents (web pages) collected from various sites of around 10k-30k chars each, I am processing them to extract relevant text as much as possible, but they are never perfect.
...
- 120
0
votes
1
answer
93
views
A way to init sentence embedding for unsupervised text clustering, better than glove wordvec?
For unsupervised text clustering, the key thing is the init embedding for text.
If we want to use deepcluster for text, the problem for text is how to get the init embedding from deep model.
BERT can ...
- 691
1
vote
0
answers
64
views
How to do embedding for nested dictionary with varying size?
I'm working on an RL task in which the agent needs have some observation. Instead using images, I want to use available information of the environment as the observation. The information regarding the ...
- 111
1
vote
1
answer
98
views
Literature on selecting specific dimensions in a word embedding vector
I am aware that the different dimensions in the word embedding represents different information and algebraic operations can be performed between two embeddings for example.
Can anyone point me to ...
- 111
1
vote
1
answer
325
views
GPU vs TPU for convolutional neural networks (NLP)
I am testing ideas on IMDB sentiment analysis task by using word embeddings + CNN approach. What could explain a significant difference in computation time in favor of GPU (~9 seconds per epoch) ...
- 111
3
votes
1
answer
155
views
How to work with different Encoding for Foreign Languages
I've got a Word Embedding File called model.txt. This contains 100 Dimensional vectors for over a million French words. These words contain accented characters such ...
- 529
1
vote
1
answer
752
views
Embedding of list of objects
I have a dataset where each sample is a list of ordered items, lets say grocery list , and a label from 6 categories . each list can have up to 120 items but the mean items is 12 items in a list.
i ...
- 293
1
vote
0
answers
17
views
Embedding layers trained on Amazon Reviews
I am working on research to perform sentiment analysis on Amazon reviews. My data is not labelled so I am now using Lexicon based sentiment analysis such as Vader. I am wondering if it is possible to ...
- 13
0
votes
1
answer
406
views
Why does English ELMo model give embeddings for non-English words?
Here's the code from my notebook:
...
- 131
0
votes
1
answer
168
views
ELMo - How does the model transfer its learning/weights on new sentences
Word2vec and Glove embeddings have the same vector representation for every word in the corpus and does not take context into consideration.
For eg:
The dog does bark at people
The bark of the tree ...
- 53
1
vote
0
answers
576
views
Extracting vectors of FastText own model to use it on a NN
I have trained my own model of fasttext using the pretrained model of English available on their website with the next code:
...
- 111
4
votes
1
answer
292
views
Why are character level models considered less effective than word level models?
I have read that character level models need more computation power than word embeddings, and this is one of the major reasons for their less effectiveness, but i got curious because the word ...
- 41
0
votes
0
answers
108
views
Using word embeddings for kaggle?
Not sure, if this is the right forum so redirect me if it wrong.
I have started on an NLP problem in kaggle. There i have word embeddings from google news, wiki, glove
in a zipped folder. I want to ...
- 1
0
votes
2
answers
150
views
Is the number of bidirectional LSTMs in encoder-decoder model equal to the maximum length of input text/characters?
I'm confused about this aspect of RNNs while trying to learn how seq2seq encoder-decoder works at https://machinelearningmastery.com/configure-encoder-decoder-model-neural-machine-translation/.
It ...
- 123
0
votes
1
answer
171
views
How to Calculate semantic similarity between video captions?
I intend to calculate the accuracy of a caption generated by comparing it to a number of reference sentences.
For example, the captions for one video are as follows:
These captions are for the same ...
- 1
2
votes
1
answer
2k
views
What information does output of [SEP] token captures in BERT?
After reading around on the web I came to understand that the output representation of the special token [CLS] captures the representation of a sentence (am I correct?).
My primary question is what ...
- 35
1
vote
1
answer
32
views
How to go about creating embeddings (especially, token to Id mapping) for categorical columns in tensorflow 2.0+?
I have a csv with both categorical and float dtypes. I want to do the following:
For each categorical column i will use pandas to compute the unique values (...
- 821
0
votes
1
answer
56
views
Predicting correct match of French to English food descriptions
I have a training and test set of food descriptions pairs (please, see example below)
First name in a pair is a name of food in French
and second word is this food description in English.
Traing set ...
- 295
1
vote
1
answer
1k
views
How to ignore vectors of zeros (i.e. paddings) in Keras?
I'm implementing a LSTM model with Keras. My dataset is composed by words and each word is an 837 long vector. I grouped the words in groups of 20 and to do this I ...
- 405
1
vote
1
answer
187
views
How to train NER LSTM on single sentence level
My documents are only a single sentence long, containing one annotation.
Sentences with the same named entity of course are similar, but not context-wise.
NER ...
- 103
1
vote
1
answer
218
views
Heterogeneous clustering with text data
I have a dataset which consists of multiple user ratings. Each rating looks similarly to:
...
- 13
1
vote
1
answer
1k
views
Problem when using Autograd with nn.Embedding in Pytorch
I am in trouble with taking derivatives of outputs logits with respect to the inputs input_ids. Here is an example of my input:
...
- 113