Questions tagged [word-embeddings]
Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.
177
questions with no upvoted or accepted answers
7
votes
3
answers
4k
views
Why is 10000 used as the denominator in Positional Encodings in the Transformer Model?
I was working through the All you need is Attention paper, and while the motivation of positional encodings makes sense and the other stackexchange answers filled me in on the motivations of the ...
7
votes
1
answer
9k
views
How is WordPiece tokenization helpful to effectively deal with rare words problem in NLP?
I have seen that NLP models such as BERT utilize WordPiece for tokenization. In WordPiece, we split the tokens like playing to play and ##ing. It is mentioned that it covers a wider spectrum of Out-Of-...
- 696
3
votes
0
answers
362
views
Text embeddings for words or very short sentences with a LLM
I tried to compute the semantic similarity between words or short sentences.
Ex : inflation vs price raising
I have tried the openai embeddings API and cosine distance but the results are very poor.
I ...
- 31
3
votes
2
answers
616
views
Gensim doc2vec error: KeyError: "word 'senseless' not in vocabulary"
I am new to machine learning and tried doc2vec on quora duplicate dataset. new_dfx has columns 'question1' and 'question2' which has preprocessed questions in each row. Following is the tagged ...
3
votes
1
answer
2k
views
Difference between NCE-Loss and InfoNCE-Loss
I started looking into word2vec and was wondering what the connection/difference between the NCE-Loss and the infoNCE-Loss is. I get the basic idea of both.
I have a hard time deriving one from ...
- 31
3
votes
0
answers
182
views
Medical NER for French language
I'm currently exploring the options to extract medical NER specifically for French language. I tried SpaCy's general French NER but it wasn't helpful to the cause (...
- 285
3
votes
1
answer
783
views
Character-level embeddings in python
I'm working on an NLP task that requires the use of character level embeddings, and I've been trying to use Spacy. However, it seems that spacy uses word-level embeddings for the word vectors, and I ...
- 163
3
votes
1
answer
204
views
How to build a symmetric similarity model on top of embeddings?
I have two equal length vectors that come out of two identical embedding layers.
I want to calculate their similarity, and I don't trust the embedding layer enough to just use dot product (e.g. it's ...
- 131
3
votes
2
answers
183
views
How to train neural word embeddings?
So I am new to Deep Learning and NLP. I have read several blog posts on medium, towardsdatascience and papers where they talk about pre-training the word embeddings in an unsupervised fashion and then ...
- 139
3
votes
1
answer
505
views
How to train millions of doc2vec embeddings using GPU?
I am trying to train a doc2vec based on user browsing history (urls tagged to user_id). I use chainer deep learning framework.
There are more than 20 millions (user_id and urls) of embeddings to ...
- 171
2
votes
1
answer
194
views
Does Word2Vec's skip-gram NNLM even produce context words?
Let me first establish what CBoW and skip-gram are supposed to do. You can skip to the next section if you think this is unnecessary.
Background
My understanding is that Word2Vec is a suite of 2 ...
- 193
2
votes
0
answers
119
views
how Can we add extra word embedding to the pytorch funnel transformer?
i was approaching NLP sequence classification problem (3 classes) using huggingface transformers (funnel-transformer/large) and tensorflow.
first i created laserembedding like this :
...
- 121
2
votes
1
answer
406
views
Range of values of BERT and other embeddings?
Are the values in all NLP models' embeddings between the range -1 to 1?
If not, what models use a different range (or decimal points)? And what could be the reason for that shift/change?
- 133
2
votes
0
answers
164
views
Ways to cluster word senses with word embeddings
I'm trying to semantically cluster polysemous words or word with different meanings in a corpus for my class study and I want to do it by word embeddings but I have no Idea how to reach to the ...
- 21
2
votes
1
answer
28
views
Types of averages when analyzing sentences
I have a list of words and their frequencies in a text corpus. So there are words like "a", "what", "some" that have really high frequencies, and other like "...
- 141
2
votes
1
answer
171
views
The embeddings of GoogleNews-vectors-negative300.bin is stemmed or not?
I will use the embedding GoogleNews-vectors-negative300.bin data to do some project. Just want to know whether it was stemmed of the words?
- 31
2
votes
0
answers
685
views
Learning words embedding for bigrams and unigrams in a corpus
I am working on a topic modeling for tweets projects. I have generated my topics using both unigrams and bigrams. Topics are defined with a mixture of both bigrams and unigrams.
Now I am planning to ...
- 191
2
votes
0
answers
277
views
How to use paraphrase_mining using sentence transformers pre-trained model
I am trying to find similarity between sentences using a pre-trained sentence-transformers model. I am trying to follow the code here - https://www.sbert.net/docs/usage/paraphrase_mining.html
In trial ...
- 121
2
votes
1
answer
28
views
Training pipelines where featurization/NLP is more expensive than backprop
I'm working on a document classification project and I'm using a neural net in tensorflow, where the features are 300-dimensional word embeddings, either from fastext or word2vec (yes I know that ...
- 499
2
votes
0
answers
104
views
More pre-trained embeddings for PyTorch Big Graph
Apart from the entire wikidata, are there any other PyTorch Big Graph pre-trained graph embeddings on smaller sized knowledge graph, like freebase-15k? I do not have the resources to build it from ...
- 121
2
votes
2
answers
196
views
Semantic Search
There is a problem we are trying to solve where we want to do semantic search on our set of data,
i.e we have a domain specific data (example: sentences talking about automobiles)
Our data is just a ...
2
votes
1
answer
118
views
Building own embedding from sequence
I have 100 sequences of the word (i.e., action for completing a task). Each of the sequences contains around 350 actions(115 unique actions but all the actions are not used in each sequence. Some of ...
2
votes
0
answers
81
views
NLP based Data Preprocessing Method to Improve Disease Name Prediction Using CRF and Word Embedding
I built a model( using CRF along bi lstm) to Predict New Disease Name/Entities from medical text data but the problem is Disease name appears only 5,6 times in 1 text file but on average text file ...
- 21
2
votes
1
answer
360
views
Predicting word from a set of words
My task is to predict relevant words based on a short description of an idea. for example "SQL is a domain-specific language used in programming and designed for managing data held in a relational ...
- 221
2
votes
1
answer
621
views
What are the equations involved in calculation of the parameters of embedding layer?
I'm trying to perform sentiment analysis on some data using keras.I'm using embedding layer and then LSTM. I know that embedding layer decreases the sparsity of the one hot encodings of the words and ...
- 41
2
votes
1
answer
256
views
Improve results using user input
I've developed a tool that retrieve the closest expressions from a database based on what the user typed. (using word embedding - a comparison is made between each expression from the database and the ...
- 21
2
votes
0
answers
470
views
Keras value error: Operands could not be broadcast with with shapes(100,100) - GRU
I am trying to use Hierarchical Attention Networks for classification of news articles using 20 newsgroup dataset that i downloaded from the internet. I came across this code of the implementation and ...
- 123
2
votes
2
answers
1k
views
Combining different features as input to Neural Network
I use two different sources of information as input to my neural model. The model takes a word as input and produces binary [1/0] output. I have represented each word by using its word embedding (1024 ...
- 249
2
votes
1
answer
2k
views
Enable Mini-batch Processing on PyTorch Word Embeddings
I am new to PyTorch and trying to create word embeddings. I started with the example below and everything works fine and it completes relatively quickly.
...
- 978
2
votes
0
answers
194
views
NER at sentence level or document level?
Should NER models (LSTM or CRF) take input training data at sentence level or paragraph level?
Let's say we have this input text, and we would like to do Named Entity Extraction:
GOP Sen. Rand ...
1
vote
0
answers
48
views
I can't get good performance from BERT
I trained NLP models. This is a subset (200 instances) of my data set of 10,000 instances:This the link of the dataset on pastebin
I compare an LSTM model with a glove model and a BERT model. I ...
- 131
1
vote
0
answers
118
views
Otimization of similarity search for multiple embeddings by creating a weighted artificial embedding
I have embeddings of text created with a BERT model. A group of these embeddings should be used to find similar embeddings corresponding to this group. I know that you can use average or max (or ...
- 113
1
vote
1
answer
41
views
Why the label is not explicitly involved in the loss function of skip-gram?
I am recently learning word embedding myself. When learning skip-gram from the paper https://arxiv.org/pdf/1310.4546.pdf[Distributed Representations of Words and Phrases and their Compositionality], I ...
- 11
1
vote
1
answer
39
views
Which pre-trained model to select to generate embeddings from shop names written in English?
Good afternoon!
I have a dataset with thousands of shop names written in English. Several shop names might belong to one business entity, for instance, shops with names "KFC 001", "WWW....
- 11
1
vote
0
answers
42
views
Is normalization of word embeddings important?
I am doing actor-critic reinforcement learning for an environment that is best represented as a "bag-of-words". For this reason, I have opted to use a single body, multi-head approach for ...
1
vote
0
answers
68
views
Machine learning with mixed variables dataset (numerical, categorical and embeddings)
I'm working on a machine learning project where I'm trying to predict the revenue of a movie.
My dataset contains mixed data types. There are numerical features (rating, number of votes, release year,....
- 111
1
vote
0
answers
415
views
What are the inputs of encoder and decoder layers of transformer architecture?
In the paper (attention is all you need), it says "embeddings" are the input of the encoding layer. As I know embeddings are the numerical representation of words which is (for example) the ...
- 91
1
vote
0
answers
42
views
Which model is better able to understand the difference that two sentences are talking about different things?
I'm currently working on the task of measuring semantic proximity between sentences. I use fasttext train _unsiupervised (skipgram) for this. I extract the sentence embeddings and then measure the ...
- 183
1
vote
0
answers
46
views
Contextual word embeddings from pretrained word2vec vectors
I would like to create word embeddings that take context into account, so the vector of the word Jaguar [animal] would be different from the word Jaguar [car brand].
As you know, word2vec only gives ...
- 11
1
vote
0
answers
242
views
How are the embedding and context matrices created and updated in word embedding?
I am struggling to understand how word embedding works, especially how the embedding matrix $W$ and context matrix $W'$ are created/updated. I understand that in the Input we may have a one-hot ...
1
vote
0
answers
49
views
How to deal with words out of the vocabulary CBOW implementation
I'm studying word2vec theory, and I decided to implement the Continuous Bag of Words model from zero.
I know the primary pipeline for this:
Preprocess a corpus: remove stopwords, lemmatization, etc.
...
- 111
1
vote
1
answer
415
views
What are different ways to reduce size of word2vec vectors file?
I am working on an application with memory constraints. We are getting vectors from python Gensim models but need to transmit copies of them to react native mobile app and potentially in-browser JS. ...
- 11
1
vote
2
answers
4k
views
How to calculate the mean average of word embedding and then compare strings using sklearn.metrics.pairwise
I am totally new to this topic, that's why I am so confused or stuck in this code for a while, but I am not sure how to solve it correctly.
My goal is to write a short text embedding using vector ...
- 11
1
vote
1
answer
227
views
How can I use Wikipedia2vec model for embedding my article named entities as 40% entities are not in a wikipedia?
I have news articles in my dataset containing named entities. I want to use the Wikipedia2vec model to encode the article's named entities. But some of the entities (around 40%) from our dataset ...
- 11
1
vote
0
answers
13
views
Bad accuracy on predicting user ratings upon his reviews
I am trying to predict user ratings based on books he rated(ratings range between 1-10). So I encoded the summary of the books and then tried to train the model with the text encoding(practically the ...
- 11
1
vote
0
answers
176
views
Should I use Pad Sequence when using Word Vectors?
I have an unbalanced text data set. I want to use word vectors to embed words. When I use pad sequence? Before or after the word vector? I tried it, after the word vector I used pad sequence but my ...
- 13
1
vote
0
answers
36
views
GloVe dot product optimized for non-comutative data whilst the operation itself being commutative
To my current knowledge, GloVe word vectors dot product are optimized to be the
w_i ⋅ w_j = log(P(ⅈ|j))
The probability being computed from a cooccurance matrix. However, dot product is a commutative ...
- 23
1
vote
0
answers
135
views
Word embedding autoencoder
I'm trying to train a word embedding autoencoder, but it either doesn't train, or trains but doesn't make predictions. I know I'm doing something wrong, so any help is greatly appreciated.
Here is my ...
- 11
1
vote
0
answers
122
views
Plotting cosine similarities in 3d space from word embeddings
I'm working on a project in which I want to estimate biases from a large corpus of newspaper articles using word2vec. Following this and this paper, biases are calculated by constructing dimension x ...
- 11
1
vote
1
answer
2k
views
Not clear about relative position bias
I've been reading the Swin Transformer paper and came across relative position bias concept. I'm not able to figure out how is it more effective than positional embeddings. I hope someone can explain ...
- 141