Questions tagged [word-embeddings]
Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.
507
questions
0
votes
0
answers
528
views
Can I average the BERT embeddings of multiple instances of the same word to get one vector representation of the word?
In the project I'm working on right now I would like to get one embedding for every unique lemma in a corpus. Could I get this by averaging the embeddings of every instance of a lemma?
For example, ...
- 1
0
votes
2
answers
662
views
Handling unknown words when making NER Models
I'm working on my custom Named Entity Recognition model that I'm making in Python's Keras lib.
I have read that I should enumerate all words that are appearing, so that I get vectorized sequences. I ...
- 199
1
vote
1
answer
2k
views
Not clear about relative position bias
I've been reading the Swin Transformer paper and came across relative position bias concept. I'm not able to figure out how is it more effective than positional embeddings. I hope someone can explain ...
- 141
0
votes
1
answer
22
views
applicability of relative similarity computation
I've computed the cosine similarity between a & b (=x) and ...
- 285
0
votes
1
answer
543
views
What is the difference between sparse and dense corpra?
I didn't got the meaning or the difference between sparse and dense corpra here in this sentence "the reason is that Skip-gram works better over sparse corpora like Twitter and NIPS, while CBOW ...
- 1
0
votes
1
answer
183
views
Getting context-word pairs for a continuous bag of words model and other confusions
Suppose I have a corpus with documents:
...
- 133
0
votes
1
answer
142
views
Question regarding training data in word2vec - skip-gram
I have a very simple question regarding the training data in word2vec. In the skip-gram implementation, the training data (if I understand it correctly) is generated as pairs of words like it's shown ...
- 101
0
votes
1
answer
116
views
Ensure trained word embeddings get high similarity with particular words
I am trying out my hand at training a Word2Vec model using gensim. I made a simple training file that basically had just one line repeated multiple times
...
- 101
0
votes
1
answer
42
views
What is the standard output of the GloVe algorithm?
When I look at the loss function of the GloVe algorithm for generating word vectors, I see that $w$ and $\tilde{w}$ are symmetric:
$$
J=\sum_{i,j=1}^Vf(X_{ij})(w_i^T\tilde{w}_j+b_i+b_j-logX_{ij})^2
$$...
- 267
1
vote
1
answer
444
views
K-Means clustering in the analysis of Word2vec embeddings
I have a yelp-review dataset. I have done a word2vector embedding on the text column of the yelp-review. I am using unsupervised leaning K-means and PCA & TSNE to visualise the data. I have got 6 ...
- 111
0
votes
1
answer
24
views
Influence of label names on the classfierier perfromance
I am building a text classifier, the labels in my training data are not just short names like "Dog" or "Cat", they are more of lengthy sentences that range from 2 words to around ...
- 103
1
vote
1
answer
198
views
How can i extract words from a single concatenated word?
I'm stuck on this problem and would love some input.
I have mulitple words such as getExtention, getPath, someWord or someword and i want to separate each concatinated words into its own words such as:...
1
vote
0
answers
242
views
Hyper parameters (window size and vector dimensions) tuning in word2vec using Grey Wolf Optimization
Using Grey wolf Optimization, I want to calculate optimal values of two hyper parameters: context window size and embedding size (vector dimensions) for word2vec skipgram model used for word embedding....
- 11
2
votes
1
answer
373
views
comparaison of two embedding vector
I am working on embedding, i would like to know which if Mean Squared error (MSE) is better to make comparison between two embedding vector than Co-sinus similarity. In which situation use one or ...
2
votes
1
answer
1k
views
Cluster words into groups of similar meaning (synonyms)
How can words be clustered into groups of similar meaning (synonyms)?
I started with pre-trained word embeddings (e.g., Google News), which is great, but not perfect - a limitation arises because the ...
- 141
1
vote
1
answer
78
views
How Sklearn-crfsuit interpret text features
As we see here, to build an NER model we can pass text features (parts of the word, pos tag, structure of the word etc.) to Sklearn-CRF. I was wondering how does this package convert the text features ...
4
votes
1
answer
2k
views
What happens when the vocab size of an embedded layer is larger than the text corpus used in training?
Full disclosure this question is based on following this tutorial: https://tinyurl.com/vmyj8rf8
I am trying to fully understand embedded layers in Keras. Imagine having a network to try and understand ...
- 237
1
vote
0
answers
546
views
How do I convert my dataframe column into vectors using word2vec?
I have the column of Categories of vary length in my DataFrame as shown below :
Categories
[Restaurants , Drinks]
[Restaurants , Drinks , Rooftop]
[Dinner]
.
.
.
...
- 229
0
votes
1
answer
550
views
Where the objective function of Skip-Gram Negative Sampling (SGNS) come from?
In the paper of "Neural Word Embedding as Implicit Matrix Factorization", there is an objective function for Skip-Gram Negative Sampling. I wanted to know where this formula come from
- 263
5
votes
2
answers
1k
views
Why is averaging the vectors required in word2vec?
While implementing word2vec using gensim by following few tutorials online, one thing that I couldn't understand is the reason why word vectors are averaged once the model is trained. Few example ...
- 163
1
vote
1
answer
268
views
BERT is running out of memory in forward pass for my dictionary
Running code from this answer, my BERT is running out for my 4k words dictionary.
I don't need to do anything with these words yet, just make embeddings for my data. So, using this exactly:
...
- 137
1
vote
1
answer
415
views
Why does word2vec try to maximize the dot product of the center word vector and context?
I am learning about the maths behind word2vec from this tutorial.
u are the embeddings for the center word and v for the context. It appears that this dot product is to be maximized. Why the context ...
- 189
0
votes
2
answers
4k
views
How to JUST represent words as embeddings by pretrained BERT?
I don't have enough data (i.e. I don't have enough texts) --- have only around 4k words in my dictionary. I need to compare given words, then I need to representate it as embedding.
After the ...
- 137
1
vote
3
answers
62
views
Building a multiclass classifier that can handle classes it has never seen?
I am given a dataset that has free-form text and a category associated with it. There are 100 different categories and 3000 records for each category. The goal is build a multiclass classification ...
- 11
2
votes
3
answers
1k
views
The reason behind using a pre-trained model?
These last month I have been studying all about word embeddings and the most known pre-trained word embeddings, Word2Vec, GloVe, FastText, etc. I have read many times how important It is to take ...
- 21
0
votes
1
answer
71
views
Word2Vec: Identifying many-to-one relationships between words
Standard introductory examples in Word2Vec, like king - queen = man - woman and tokyo - japan = london - uk, involve one-to-one ...
1
vote
0
answers
42
views
When are subword ngrams trained in fasttext? (Enriching Word Vectors with Subword Information)
when is the training for subword ngrams done? is it done simultaneously as when the word representation are trained? or is it done at the end, after word representations are created?
fasttext ...
- 667
3
votes
1
answer
2k
views
Difference between NCE-Loss and InfoNCE-Loss
I started looking into word2vec and was wondering what the connection/difference between the NCE-Loss and the infoNCE-Loss is. I get the basic idea of both.
I have a hard time deriving one from ...
- 31
1
vote
1
answer
2k
views
Sentence embeddings with LSTM to classify the sentences is not working
I am trying to build LSTM NN to classify the sentences. I have seen many examples where sentences are converted to word vectors using glove, word2Vec and so on here is an example of it. This solution ...
- 159
1
vote
1
answer
57
views
How can I convert my predictions to text after predicting using RNN?
I'm building PoS tagger for our language. I give tokens to the words and tags using Tokenizer(). Functions for word and tag are different.
...
- 11
0
votes
1
answer
1k
views
Where can I find documentation or paper mentioning pre-trained distilbert-base-nli-mean-tokens model?
I am trying to find more information about pre-trained model distilbert-base-nli-mean-tokens. Can someone please point me to it's paper or documentation? Is it ...
- 2,041
2
votes
1
answer
168
views
The embeddings of GoogleNews-vectors-negative300.bin is stemmed or not?
I will use the embedding GoogleNews-vectors-negative300.bin data to do some project. Just want to know whether it was stemmed of the words?
- 31
1
vote
1
answer
50
views
Preparing training data for NLP machine learning task
I have the natural language sentences as follows:
This is a black chair. It is next to the table.
Each phrase that represents an object is annotated with an object ...
- 155
0
votes
1
answer
310
views
Word embedding for a single word
I want to tackle email similarity with a word embedding approach, not an expert in embedding text but it is possible to embed emails where similar emails have similar vectors? is that okay?. I know ...
- 111
4
votes
2
answers
420
views
Can we use embeddings or latent vectors for a recommender system?
I'm having a hard time understanding why people use any vector they find as a candidate for a recommender system.
In my mind, a recommender system requires a space where distance represents similarity....
- 267
3
votes
0
answers
180
views
Medical NER for French language
I'm currently exploring the options to extract medical NER specifically for French language. I tried SpaCy's general French NER but it wasn't helpful to the cause (...
- 285
1
vote
0
answers
51
views
Self-supervised learning for automatic labeling of data using LDA and Word2Vec
I am trying to implement this paper A Brand-New Look at You: Predicting Brand Personality in Social Media Networks with Machine Learning for labeling Twitter data of brands with a corresponding brand ...
- 213
1
vote
0
answers
363
views
How does Google's Universal Sentence Encoder deal with out-of-vocabulary terms?
It seems to output embeddings even for random jibberish, and the similarity is even high for this particular pair of jibberish.
...
- 33
2
votes
0
answers
681
views
Learning words embedding for bigrams and unigrams in a corpus
I am working on a topic modeling for tweets projects. I have generated my topics using both unigrams and bigrams. Topics are defined with a mixture of both bigrams and unigrams.
Now I am planning to ...
- 191
1
vote
1
answer
139
views
What is a good approach for embedding both textual and spatial features for document classification?
I am working on a document classifier that can perform the classification based on the document structure as well. My plan is to get the word embedding as well as the word coordinates and somehow ...
- 111
1
vote
1
answer
137
views
How to find coherence between a large number of sentences
I have a list of sentences returned as a result of a document search algorithm. I want to determine if the results returned are semantically close/similar/coherent using some sort of metric. For a ...
- 53
3
votes
3
answers
3k
views
Dot product for similarity in word to vector computation in NLP
In NLP while computing word to vector we try to maximize log(P(o|c)). Where P(o|c) is probability that o is outside word, given that c is center word.
Uo is word vector for outside word
Vc is word ...
- 129
0
votes
0
answers
323
views
Adding extra feature to word2vec embeddings for use in classification
I have a word2vec model that will transform my text into vectors, and I need to use them to classify the text. My data is essentially a timeseries of chat messages, so I think that the message ...
- 1
5
votes
2
answers
2k
views
how to run bert's pretrained model word embeddings faster?
I'm trying to get word embeddings for clinical data using microsoft/pubmedbert.
I have 3.6 million text rows. Converting texts to vectors for 10k rows takes around 30 minutes. So for 3.6 million rows, ...
- 138
6
votes
2
answers
1k
views
Word2Vec: Why do some dimensions of an embedding have an interpretation, and why does addition/subtraction of embedding vectors work?
I'm reading about Word2Vec from this source: http://jalammar.github.io/illustrated-word2vec/. Below is the heatmap of the embeddings for various words. In the source, it's claimed that we can get an ...
- 299
1
vote
1
answer
403
views
Context Based Embeddings vs character based embeddings vs word based embeddings
I am working on a problem that uses English alphabets in the text but the language is not English. Its a mixture of English and different language text. But all words are written using English ...
- 113
1
vote
0
answers
15
views
Very infrequent values in embedding layers
I have a categorical input that is very imbalanced. 90% of the values are either A or B and frequencies for C, D, E, F, etc are as little as 1.
I am using an embedding layer for this input and the ...
- 105
1
vote
1
answer
530
views
Continuous Bag Of Words (CBOW) network architecture?
Looking into word2vec like embeddings I found this exercise on PyTorch's website which prompts the reader to implement a CBOW network in PyTorch.
My question is about the architecture to implement ...
- 11
2
votes
1
answer
1k
views
How to test the quality of a word embedding?
I have trained a word2vec model using GenSim 4.
The problem is that my corpus is quite small.
How can I test the quality of the word embeddings I have obtained?
Is there some standard measures to do ...
- 196
0
votes
1
answer
58
views
Vector representation of documents for text classification
I'm looking for proper method of document embeddings. I know that doc2vec will give me the vector representations for given corpus, but how do I embed new documents? I need to train neural network ...
