Questions tagged [word-embeddings]
Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.
507
questions
1
vote
0
answers
46
views
Contextual word embeddings from pretrained word2vec vectors
I would like to create word embeddings that take context into account, so the vector of the word Jaguar [animal] would be different from the word Jaguar [car brand].
As you know, word2vec only gives ...
- 11
1
vote
1
answer
138
views
Is there a sensible notion of 'character embeddings'?
There are several popular word embeddings available (e.g., Fasttext and GloVe); In short, those embeddings are a tool to encode words along with a sensible notion of semantics attached to those words (...
- 123
1
vote
0
answers
233
views
How are the embedding and context matrices created and updated in word embedding?
I am struggling to understand how word embedding works, especially how the embedding matrix $W$ and context matrix $W'$ are created/updated. I understand that in the Input we may have a one-hot ...
6
votes
1
answer
2k
views
Sum vs mean of word-embeddings for sentence similarity
So, say I have the following sentences
["The dog says woof", "a king leads the country", "an apple is red"]
I can embed each word using an ...
- 460
1
vote
1
answer
123
views
Why is Word2vec regarded as a neural embedding?
In the skip-gram model, the probability that a word $w$ is part of the set of context words $\{w_o^{(i)}\}$ $(i= 1:m)$ where $m$ is the context window around the central word, is given by:
$$p(w_o | ...
- 43
1
vote
1
answer
490
views
Why we need to 'train word2vec' when word2vec itself is said to be 'pretrained'?
I get really confused on why we need to 'train word2vec' when word2vec itself is said to be 'pretrained'? I searched for word2vec pretrained embedding, thinking i can get a mapping table directly ...
- 411
1
vote
1
answer
44
views
Application of bag-of-ngrams in feature engineering of texts
I've got few questions about the application of bag-of-ngrams in feature engineering of texts:
How to (or can we?) perform word2vec on bag-of-ngrams?
As the feature space of bag of n-gram increases ...
- 411
1
vote
1
answer
636
views
Do we perform text embedding before or after train-test splitting?
Do we perform text embedding before or after train-test splitting?
I know that for encoding variables, usually done after the split. However, I'm not sure if that's also the case for text processing?
- 411
1
vote
0
answers
49
views
How to deal with words out of the vocabulary CBOW implementation
I'm studying word2vec theory, and I decided to implement the Continuous Bag of Words model from zero.
I know the primary pipeline for this:
Preprocess a corpus: remove stopwords, lemmatization, etc.
...
- 111
2
votes
1
answer
2k
views
Measuring similarity from massive embedded vectors
I am given a set of 10,000 journal articles, with their corresponding 100th-dimension embedded vectors. (The way they are embedded is unknown, but I'm guessing it is ...
- 23
1
vote
1
answer
452
views
What is the meaning of two embedding layers in a row?
I've noticed in one deep pre-trained textual neural network that there are two embedding layers in the beginning and I don't quite understand why there are two of them. As far as I understand (correct ...
- 157
1
vote
1
answer
411
views
What are different ways to reduce size of word2vec vectors file?
I am working on an application with memory constraints. We are getting vectors from python Gensim models but need to transmit copies of them to react native mobile app and potentially in-browser JS. ...
- 11
5
votes
2
answers
5k
views
What are the exact differences between Word Embedding and Word Vectorization?
I am learning NLP. I have tried to figure out the exact difference between Word Embedding and Word Vectorization. However, seems like some articles use these words interchangeably. But I think there ...
- 53
1
vote
1
answer
156
views
Latent space vs Embedding space | Are they same?
I am going through variational autoencoders and it is mentioned that:
continuity (two close points in the latent space should not give two
completely different contents once decoded) and completeness ...
- 155
5
votes
3
answers
1k
views
Best way to vectorise names and addresses for similarity searching?
I have a large dataset of around 9 million people with names and addresses. Given quirks of the process used to get the data it is highly likely that a person is in the dataset more than once, with ...
- 237
0
votes
3
answers
334
views
Does word2vec fail for window size equal to sentence size
Will word2vec fail if sentences contain only similar words, or in other words, if the window size is equal to the sentence size? I suppose this question boils down to whether word to vec considers ...
- 53
0
votes
1
answer
17
views
How do companies handle changing natural language
I am assuming large social medias like Twitter handle hashtags using some sort of embedding, so that similar tweets can be found or suggested. Maybe this is a bad assumption- maybe someone can clarify....
- 53
0
votes
2
answers
673
views
Word Embedding Dimensions Reduction
In my NLP task, I use Glove to get each word embedding, Glove gives 50 float numbers as an embedding for every word in my sentence, my corpus is large, and the resulted model is also large to fit my ...
- 118
0
votes
1
answer
733
views
How to compute sentence embedding from word2vec model?
I am new to NLP and I'm trying to perform embedding for a clustering problem. I have created the word2vec model using Python's gensim library, but I am wondering ...
- 111
1
vote
1
answer
835
views
Positional encoding without input embedding
Does it make sense to use a positional encoding in attention when the input tokens do not go through an embedding layer?
In NLP models, the embedding maps a word to real numbers. ...
- 113
0
votes
1
answer
1k
views
What are the differences between bert embedding and flair embedding
I read about BERT embedding model and FLAIR embedding model, and I'm not sure I can tell what are the differences between them ?
...
- 363
0
votes
2
answers
75
views
1
vote
2
answers
2k
views
Why does averaging word embedding vectors (exctracted from the NN embedding layer) work to represent sentences?
I'm puzzling to understand why the method of averaging word embeddings works in order to obtain sentence embedding, in particular considering the exercize of this post How to obtain vector ...
0
votes
0
answers
121
views
Conceptual question about cosine similarity and clustering algorithms for word embeddings
Is the following statement true? https://stats.stackexchange.com/q/256778
The value of cosine similarity between two terms itself is not indicator whether they are similar or not.
If yes then how is ...
1
vote
1
answer
101
views
does ValueError: 'rat' is not in list means not exist in tokenizer
Does this error means that the word doesn't exist in the tokenizer
return sent.split(" ").index(word)
ValueError: 'rat' is not in list
the code sequences ...
- 17
1
vote
1
answer
2k
views
How can i get the vector of word using BERT?
I need to get word-vectors using BERT and got this function that i think it should be the one i need
...
- 113
2
votes
1
answer
2k
views
Custom Named-Entity Recognition (NER) in product titles using deep learning
I am new to machine learning and Natural Language Processing (NLP). I am trying to identify which brand, product name, dimension, color, ... a product has from its product title. That is, from
'Sony ...
- 23
1
vote
2
answers
4k
views
How to calculate the mean average of word embedding and then compare strings using sklearn.metrics.pairwise
I am totally new to this topic, that's why I am so confused or stuck in this code for a while, but I am not sure how to solve it correctly.
My goal is to write a short text embedding using vector ...
- 11
1
vote
1
answer
217
views
How can I use Wikipedia2vec model for embedding my article named entities as 40% entities are not in a wikipedia?
I have news articles in my dataset containing named entities. I want to use the Wikipedia2vec model to encode the article's named entities. But some of the entities (around 40%) from our dataset ...
- 11
3
votes
1
answer
545
views
Which method is more suitable? overfitting of traning data or low accuracy?
Recently, I tested two methods after embedding in my data, using Keras.
Convolution after embedding
Maxpooling after embedding
The first method's loss and validation loss are like,
The second ...
- 93
3
votes
1
answer
154
views
Question on embedding similarity / nearest neighbor methods [SCANN Paper]
Question on embedding similarity / nearest neighbor methods:
In https://arxiv.org/abs/2112.04426 the DeepMind team writes:
For a database of T elements, we can query the approximate nearest neighbors ...
- 2,460
0
votes
1
answer
595
views
Word-level text generation with word embeddings – outputting a word vector instead of a probability distribution
I am currently researching the topic of text generation for my university project. I decided (ofc) to go with a RNN getting a sequence of tokens as input with a target of predicting the next token ...
- 101
1
vote
1
answer
449
views
When to use GloVe vocabulary vs. building a vocabulary from the training data?
While studying some (pytorch) examples that use pretrained GloVe vectors I came across two variants:
Use the vocabulary of the GloVe vectors and thus initialize the embedding layer with the ...
1
vote
0
answers
13
views
Bad accuracy on predicting user ratings upon his reviews
I am trying to predict user ratings based on books he rated(ratings range between 1-10). So I encoded the summary of the books and then tried to train the model with the text encoding(practically the ...
- 11
1
vote
0
answers
176
views
Should I use Pad Sequence when using Word Vectors?
I have an unbalanced text data set. I want to use word vectors to embed words. When I use pad sequence? Before or after the word vector? I tried it, after the word vector I used pad sequence but my ...
- 13
3
votes
2
answers
1k
views
How to get word embedding in CBOW?
I find clear explanations for skip-gram model. We take the output weight matrix, multiply it with the one-hot vector of the word we want to get the embedding.
How does it work in case of CBOW?
I know ...
- 458
1
vote
0
answers
36
views
GloVe dot product optimized for non-comutative data whilst the operation itself being commutative
To my current knowledge, GloVe word vectors dot product are optimized to be the
w_i ⋅ w_j = log(P(ⅈ|j))
The probability being computed from a cooccurance matrix. However, dot product is a commutative ...
- 23
1
vote
2
answers
875
views
How to obtain vector representation of phrases using the embedding layer and do PCA with it
I am trying to understand from both a conceptual and a Python code point of view, how to represent phrases that are present in a corpus (that is used to train a neural network to classify phrases) as ...
2
votes
2
answers
615
views
How to examine if a Doc2Vec model is sufficiently trained?
I started experimenting with gensim's Doc2Vec for sentiment analysis. For the training of the embedding itself, I have seen examples using a reduced learning rate with a few 10s or even a few hundred ...
- 131
2
votes
0
answers
161
views
Ways to cluster word senses with word embeddings
I'm trying to semantically cluster polysemous words or word with different meanings in a corpus for my class study and I want to do it by word embeddings but I have no Idea how to reach to the ...
- 21
2
votes
1
answer
103
views
Using word embeddings as features in classification algorithms?
I see there are ways to combine word vectors to form documents by taking averages or weighted averages. However, as a result of averaging there is a loss of information. Are there ways to retain the ...
- 149
0
votes
1
answer
206
views
Training fasttext on your own corpus
I want to train fasttext on my own corpus. However, I have a small question before continuing. Do I need each sentences as a different item in corpus or can I have many sentences as one item?
For ...
- 113
1
vote
0
answers
134
views
Word embedding autoencoder
I'm trying to train a word embedding autoencoder, but it either doesn't train, or trains but doesn't make predictions. I know I'm doing something wrong, so any help is greatly appreciated.
Here is my ...
- 11
3
votes
2
answers
465
views
How can word2vec or BERT be used for previously unseen words
Is there any way to modify word2vec or BERT to extend finding out embeddings for words that were not in the training data? My data is extremely domain-specific and I don't really expect pre-trained ...
- 47
1
vote
1
answer
180
views
For text classification, would a BoW or Word Embeddings based model ever be better than a Language Model?
I've done a bit of research, with this being the best as far as objectively measuring quality, but wanted to ask from a theoretical perspective if BoW-based models (e.g. using TF-IDF) or word ...
- 111
2
votes
1
answer
28
views
Types of averages when analyzing sentences
I have a list of words and their frequencies in a text corpus. So there are words like "a", "what", "some" that have really high frequencies, and other like "...
- 141
1
vote
0
answers
122
views
Plotting cosine similarities in 3d space from word embeddings
I'm working on a project in which I want to estimate biases from a large corpus of newspaper articles using word2vec. Following this and this paper, biases are calculated by constructing dimension x ...
- 11
2
votes
1
answer
3k
views
Is it possible to add new vocabulary to BERT's tokenizer when fine-tuning?
I want to fine-tune BERT by training it on a domain dataset of my own. The domain is specific and includes many terms that probably weren't included in the original dataset BERT was trained on. I know ...
- 21
0
votes
1
answer
523
views
Word Embeddings fastText in 50 dimension
Is there a fastText embedding in 50 dimensions? I'm aware of GloVe embedding is dimensions (50, 100, 200, 300) dimensions. I am trying to sentiment analysis with a very small dataset.
If there is ...
- 1
2
votes
2
answers
215
views
Why are words represented by frequency counts before embedding?
Before getting vector representations of words by embedding, the words are mapped to numbers. These numbers are chosen to be the frequency of that word in the dataset. Why does this convention exist? ...
- 23