Questions tagged [word-embeddings]
Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.
507
questions
1
vote
1
answer
268
views
BERT is running out of memory in forward pass for my dictionary
Running code from this answer, my BERT is running out for my 4k words dictionary.
I don't need to do anything with these words yet, just make embeddings for my data. So, using this exactly:
...
CommunityBot
- 1
0
votes
1
answer
183
views
Getting context-word pairs for a continuous bag of words model and other confusions
Suppose I have a corpus with documents:
...
CommunityBot
- 1
2
votes
1
answer
610
views
What are the equations involved in calculation of the parameters of embedding layer?
I'm trying to perform sentiment analysis on some data using keras.I'm using embedding layer and then LSTM. I know that embedding layer decreases the sparsity of the one hot encodings of the words and ...
CommunityBot
- 1
1
vote
1
answer
139
views
What is a good approach for embedding both textual and spatial features for document classification?
I am working on a document classifier that can perform the classification based on the document structure as well. My plan is to get the word embedding as well as the word coordinates and somehow ...
CommunityBot
- 1
1
vote
1
answer
47
views
How does Bert masked language modelling task make sense if half the time the next sentence is wrong context in the sequence passed through the encoder
Bert has two types of tasks that it uses to learn contextual word embeddings:
Masked word prediction
Next sentence prediction
I have read the paper and even there the training details are a little ...
- 821
0
votes
1
answer
543
views
What is the difference between sparse and dense corpra?
I didn't got the meaning or the difference between sparse and dense corpra here in this sentence "the reason is that Skip-gram works better over sparse corpora like Twitter and NIPS, while CBOW ...
CommunityBot
- 1
1
vote
1
answer
97
views
Literature on selecting specific dimensions in a word embedding vector
I am aware that the different dimensions in the word embedding represents different information and algebraic operations can be performed between two embeddings for example.
Can anyone point me to ...
CommunityBot
- 1
0
votes
0
answers
31
views
How was the word2vec model trained?
Let's take the CBOW (continuous bag of words) model as the example.
Suppose that, there are $c$ context words, each of which is a one-hot encoding vector. So the total number of elements of input ...
- 46
2
votes
1
answer
116
views
Building own embedding from sequence
I have 100 sequences of the word (i.e., action for completing a task). Each of the sequences contains around 350 actions(115 unique actions but all the actions are not used in each sequence. Some of ...
CommunityBot
- 1
2
votes
2
answers
1k
views
Combining different features as input to Neural Network
I use two different sources of information as input to my neural model. The model takes a word as input and produces binary [1/0] output. I have represented each word by using its word embedding (1024 ...
CommunityBot
- 1
0
votes
0
answers
10
views
For a fine-tuning a transformer to type like a specific person, should I use sentence semantic embeddings or word semantic embeddings
I'm not clear on the pros and cons of each one for this particular task. Is there even a meaningful difference?
My guess is using semantic embeddings for words will be better in nearly all cases ...
0
votes
1
answer
134
views
Doubt in ELMO, BERT, Word2Vec
I read an answer on Quora where a NLP Practioner stated that using ELMO and BERT embeddings as input to LSTM or some RNN will defeat the purpose of ELMo and BERT. I am not sure I agree with the above ...
CommunityBot
- 1
0
votes
1
answer
80
views
weights of coocurrence matrix in glove
I was studying the theory behind glove and was checking out some implementations of it. Before passing the data to its neural networks, I noticed that the weights of the co-occurrence matrix aren't ...
CommunityBot
- 1
1
vote
1
answer
2k
views
Not clear about relative position bias
I've been reading the Swin Transformer paper and came across relative position bias concept. I'm not able to figure out how is it more effective than positional embeddings. I hope someone can explain ...
CommunityBot
- 1
1
vote
1
answer
411
views
What are different ways to reduce size of word2vec vectors file?
I am working on an application with memory constraints. We are getting vectors from python Gensim models but need to transmit copies of them to react native mobile app and potentially in-browser JS. ...
CommunityBot
- 1
0
votes
1
answer
206
views
Training fasttext on your own corpus
I want to train fasttext on my own corpus. However, I have a small question before continuing. Do I need each sentences as a different item in corpus or can I have many sentences as one item?
For ...
CommunityBot
- 1
2
votes
1
answer
183
views
Does Word2Vec's skip-gram NNLM even produce context words?
Let me first establish what CBoW and skip-gram are supposed to do. You can skip to the next section if you think this is unnecessary.
Background
My understanding is that Word2Vec is a suite of 2 ...
CommunityBot
- 1
0
votes
1
answer
30
views
What does maximize average log probability mean?
In the word2vec paper (https://arxiv.org/pdf/1310.4546.pdf) that introduces the skip-gram algorithm we encounter this phrase:
which says that we maximize the average log probability.
Can someone help ...
- 23.8k
3
votes
1
answer
776
views
Character-level embeddings in python
I'm working on an NLP task that requires the use of character level embeddings, and I've been trying to use Spacy. However, it seems that spacy uses word-level embeddings for the word vectors, and I ...
CommunityBot
- 1
1
vote
1
answer
52
views
A question about contextual embeddings in the decoder only transformer architecture (gpt)
I am reading up on the decoder only architecture
Relevant excerpts:
We can use any model that maps token sequences into contextual embeddings (e.g., LSTMs, Transformers):
$$\phi : V^L \to R^{d \times ...
- 23.8k
0
votes
0
answers
23
views
Is openAI text generation models an extension of embedding models?
we can creating embeddings using below code
...
- 167
0
votes
1
answer
171
views
How to Calculate semantic similarity between video captions?
I intend to calculate the accuracy of a caption generated by comparing it to a number of reference sentences.
For example, the captions for one video are as follows:
These captions are for the same ...
CommunityBot
- 1
0
votes
1
answer
25
views
How to use location information as feature?
I have a location feature in a dataset. Some examples are: London, Uk; Sheefield Town, Ohio; UK ; North Carolina. etc. How to encode them into features? Is there any word embeddings suitable for such ...
- 1,015
0
votes
2
answers
662
views
Handling unknown words when making NER Models
I'm working on my custom Named Entity Recognition model that I'm making in Python's Keras lib.
I have read that I should enumerate all words that are appearing, so that I get vectorized sequences. I ...
CommunityBot
- 1
2
votes
2
answers
195
views
Semantic Search
There is a problem we are trying to solve where we want to do semantic search on our set of data,
i.e we have a domain specific data (example: sentences talking about automobiles)
Our data is just a ...
CommunityBot
- 1
0
votes
1
answer
93
views
A way to init sentence embedding for unsupervised text clustering, better than glove wordvec?
For unsupervised text clustering, the key thing is the init embedding for text.
If we want to use deepcluster for text, the problem for text is how to get the init embedding from deep model.
BERT can ...
CommunityBot
- 1
0
votes
1
answer
252
views
how to train custom word2vec embeddings to find related articles?
I am beginner in machine learning. My project is to make search engine based on AI which shows related articles when we search on website. For this i decided to train my own embedding.
I found two ...
CommunityBot
- 1
1
vote
1
answer
186
views
How to train NER LSTM on single sentence level
My documents are only a single sentence long, containing one annotation.
Sentences with the same named entity of course are similar, but not context-wise.
NER ...
CommunityBot
- 1
0
votes
1
answer
142
views
Question regarding training data in word2vec - skip-gram
I have a very simple question regarding the training data in word2vec. In the skip-gram implementation, the training data (if I understand it correctly) is generated as pairs of words like it's shown ...
CommunityBot
- 1
3
votes
1
answer
502
views
How to train millions of doc2vec embeddings using GPU?
I am trying to train a doc2vec based on user browsing history (urls tagged to user_id). I use chainer deep learning framework.
There are more than 20 millions (user_id and urls) of embeddings to ...
CommunityBot
- 1
1
vote
1
answer
795
views
Training Word2Vec with names instead of sentences
I have scientific database with articles and coauthors.
using this database I am training word2vec model on co-authors.
Use use case here is to disambiguate authors.
I was wondering my approach here ...
CommunityBot
- 1
0
votes
2
answers
150
views
Is the number of bidirectional LSTMs in encoder-decoder model equal to the maximum length of input text/characters?
I'm confused about this aspect of RNNs while trying to learn how seq2seq encoder-decoder works at https://machinelearningmastery.com/configure-encoder-decoder-model-neural-machine-translation/.
It ...
CommunityBot
- 1
7
votes
1
answer
9k
views
How is WordPiece tokenization helpful to effectively deal with rare words problem in NLP?
I have seen that NLP models such as BERT utilize WordPiece for tokenization. In WordPiece, we split the tokens like playing to play and ##ing. It is mentioned that it covers a wider spectrum of Out-Of-...
CommunityBot
- 1
0
votes
0
answers
197
views
How to get Llama-2 Rotary Embeddings?
I want to get the Llama-2 rotary embeddings. I do print(model) and get the following output:
In the picture I highlight the rotary embeddings.
How can get the ...
- 101
3
votes
2
answers
181
views
How to train neural word embeddings?
So I am new to Deep Learning and NLP. I have read several blog posts on medium, towardsdatascience and papers where they talk about pre-training the word embeddings in an unsupervised fashion and then ...
CommunityBot
- 1
0
votes
1
answer
36
views
How do we get output layer in skip-gram?
Could you please explain how do we get output layer in this architecture (vectors [0.2, 0.8, -1.4, 1.2] and ...
- 11
2
votes
1
answer
359
views
Predicting word from a set of words
My task is to predict relevant words based on a short description of an idea. for example "SQL is a domain-specific language used in programming and designed for managing data held in a relational ...
CommunityBot
- 1
0
votes
1
answer
116
views
Ensure trained word embeddings get high similarity with particular words
I am trying out my hand at training a Word2Vec model using gensim. I made a simple training file that basically had just one line repeated multiple times
...
CommunityBot
- 1
0
votes
1
answer
22
views
more insights about Word2Vec implementation
As we know Word2Vec is non-contextual embedding (at word level). As per my knowledge, BOW is statistical embedding technique (word level).
we can perform Word2Vec embedding in two approaches: 1. CBOW. ...
- 11
0
votes
1
answer
24
views
Appropriate input size for nn.Embedding
I’m quite new to using Pytorch and deep learning. What size of unique categories of a categorical variable is appropriate for applying the nn.Embedding ideally (best practices)? for example, if a ...
- 11
0
votes
2
answers
32
views
building embeddings for Phrases from scratch
I have a datadet with many phrases which I would like to embed them from scratch. I dont want the cosine of the words in order to get a phrase embedding, this is because the phrases may appear in a ...
- 11
0
votes
1
answer
595
views
Word-level text generation with word embeddings – outputting a word vector instead of a probability distribution
I am currently researching the topic of text generation for my university project. I decided (ofc) to go with a RNN getting a sequence of tokens as input with a target of predicting the next token ...
CommunityBot
- 1
1
vote
1
answer
2k
views
what is sentence embeding and how to do sentence embedding for a sentence and how to use word embedding to create a sentence embedding?
What is sentence embedding? How would you do sentence embedding for a sentence like: "How old are you?" How do you use word embedding to create a sentence ...
CommunityBot
- 1
1
vote
1
answer
73
views
Initializing weights that are a pointwise product of multiple variables
In two-layer perceptrons that slide across words of text, such as word2vec and fastText, hidden layer heights may be a product of two random variables such as positional embeddings and word embeddings ...
CommunityBot
- 1
0
votes
0
answers
26
views
"text" parameter in pinecone call from langchain
In this tutorial, I do not understand what "text" refers to
vectorstore = Pinecone(index, embeddings.embed_query, "text")
Could you please help?...
2
votes
1
answer
256
views
Improve results using user input
I've developed a tool that retrieve the closest expressions from a database based on what the user typed. (using word embedding - a comparison is made between each expression from the database and the ...
CommunityBot
- 1
3
votes
1
answer
199
views
How to build a symmetric similarity model on top of embeddings?
I have two equal length vectors that come out of two identical embedding layers.
I want to calculate their similarity, and I don't trust the embedding layer enough to just use dot product (e.g. it's ...
CommunityBot
- 1
2
votes
1
answer
168
views
The embeddings of GoogleNews-vectors-negative300.bin is stemmed or not?
I will use the embedding GoogleNews-vectors-negative300.bin data to do some project. Just want to know whether it was stemmed of the words?
CommunityBot
- 1
0
votes
1
answer
41
views
Some fundamental questions about Transformer
In the Transformer framework, a token as an input (time = $t$) $y^t$ is given by a sum of the original embedding of the token $x^t$ plus, a position embedding factor $v^t$, i.e.,
\begin{align}
y^t = x^...
- 101
1
vote
2
answers
3k
views
Dealing with multiple distinct-value categorical variables
So, I've got a dataset with almost all of its columns are categorical variables. Problem is that most of the categorical variables have so many distinct values.
For instance, one column have more ...
CommunityBot
- 1