Questions tagged [word-embeddings]
Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.
507
questions
1
vote
1
answer
48
views
How does Bert masked language modelling task make sense if half the time the next sentence is wrong context in the sequence passed through the encoder
Bert has two types of tasks that it uses to learn contextual word embeddings:
Masked word prediction
Next sentence prediction
I have read the paper and even there the training details are a little ...
- 821
0
votes
0
answers
31
views
How was the word2vec model trained?
Let's take the CBOW (continuous bag of words) model as the example.
Suppose that, there are $c$ context words, each of which is a one-hot encoding vector. So the total number of elements of input ...
- 46
0
votes
0
answers
10
views
For a fine-tuning a transformer to type like a specific person, should I use sentence semantic embeddings or word semantic embeddings
I'm not clear on the pros and cons of each one for this particular task. Is there even a meaningful difference?
My guess is using semantic embeddings for words will be better in nearly all cases ...
0
votes
1
answer
30
views
What does maximize average log probability mean?
In the word2vec paper (https://arxiv.org/pdf/1310.4546.pdf) that introduces the skip-gram algorithm we encounter this phrase:
which says that we maximize the average log probability.
Can someone help ...
- 198
0
votes
0
answers
24
views
Is openAI text generation models an extension of embedding models?
we can creating embeddings using below code
...
- 167
1
vote
1
answer
56
views
A question about contextual embeddings in the decoder only transformer architecture (gpt)
I am reading up on the decoder only architecture
Relevant excerpts:
We can use any model that maps token sequences into contextual embeddings (e.g., LSTMs, Transformers):
$$\phi : V^L \to R^{d \times ...
- 821
0
votes
1
answer
25
views
How to use location information as feature?
I have a location feature in a dataset. Some examples are: London, Uk; Sheefield Town, Ohio; UK ; North Carolina. etc. How to encode them into features? Is there any word embeddings suitable for such ...
- 101
0
votes
0
answers
204
views
How to get Llama-2 Rotary Embeddings?
I want to get the Llama-2 rotary embeddings. I do print(model) and get the following output:
In the picture I highlight the rotary embeddings.
How can get the ...
- 101
0
votes
0
answers
27
views
"text" parameter in pinecone call from langchain
In this tutorial, I do not understand what "text" refers to
vectorstore = Pinecone(index, embeddings.embed_query, "text")
Could you please help?...
0
votes
1
answer
41
views
Some fundamental questions about Transformer
In the Transformer framework, a token as an input (time = $t$) $y^t$ is given by a sum of the original embedding of the token $x^t$ plus, a position embedding factor $v^t$, i.e.,
\begin{align}
y^t = x^...
- 101
0
votes
1
answer
28
views
Appropriate input size for nn.Embedding
I’m quite new to using Pytorch and deep learning. What size of unique categories of a categorical variable is appropriate for applying the nn.Embedding ideally (best practices)? for example, if a ...
0
votes
0
answers
60
views
Sum of vector sentence embeddings vs. paragraph embedding
I have been experimenting with the all-MiniLM-L6-v2 model for computing 384-dimensional vector embeddings for text paragraphs. The following code compares the embedding computed for a paragraph with ...
- 141
0
votes
1
answer
381
views
Training embeddings on own dataset
In my project I follow the retrieval augmented generation (RAG) approach.
I want to create embeddings for my own dataset and use it in combination with llama-2.
In the dataset are german annual ...
- 101
3
votes
1
answer
80
views
Why do semantically different words produce similar embeddings?
I am comparing words in HuggingFace web UI using e5-small-v2, one of the best vector embedding models:
Assuming that the scores are in the range from 0 to 1, how ...
- 141
2
votes
2
answers
268
views
Why cant we use normalise position encodings instead of the cos and sine encodings used in the Transformer paper?
I'm working with Transformer models for sequence-to-sequence tasks and I'm trying to fully understand the use of positional encodings in these models.
In the original "Attention is All You Need&...
0
votes
2
answers
531
views
Recommended way to embed a text thousands of tokens long?
I've split the text up sections each 512 tokens long and created embeddings for each of them.
I want to combine them into 1 embedding for the full text. How do I do that? Is this even recommended?
...
- 268
0
votes
2
answers
32
views
building embeddings for Phrases from scratch
I have a datadet with many phrases which I would like to embed them from scratch. I dont want the cosine of the words in order to get a phrase embedding, this is because the phrases may appear in a ...
0
votes
0
answers
30
views
Imputing Missing Text Categories in Python Using Word Embeddings/Machine Learning/LLM
I'm working on a dataset where each row represents an entity with several attributes. The dataset includes fields such as ...
- 101
1
vote
1
answer
25
views
word embeddings, what are contextual word embeddings
I m a researcher, a computational linguist and I m working on word embeddings, trying to understand them.
What are contextual embeddings and how could I start working and building on my own?
Thank you ...
0
votes
1
answer
38
views
Word embeddings
I m looking into word embedding and I would like to ask if I could train words or sentences in two layers. And if I wanted that one layer is more important, how could I calculate it? For example ...
1
vote
1
answer
28
views
ELMO embeddings
Could somebody tell me how does elmo work?
Is it good for phrase embedding too?
I m looking for phrase embeddings.
Thank You in advance.
0
votes
1
answer
23
views
more insights about Word2Vec implementation
As we know Word2Vec is non-contextual embedding (at word level). As per my knowledge, BOW is statistical embedding technique (word level).
we can perform Word2Vec embedding in two approaches: 1. CBOW. ...
- 57
0
votes
1
answer
49
views
What do averaged word vectors represent?
Assume you have high-dimensional word embeddings (d > 100) for a large number of words (|V| > 100,000) calculated over a huge non-specialized natural language corpus. Assume you have taken the ...
0
votes
1
answer
36
views
How do we get output layer in skip-gram?
Could you please explain how do we get output layer in this architecture (vectors [0.2, 0.8, -1.4, 1.2] and ...
- 101
1
vote
0
answers
47
views
I can't get good performance from BERT
I trained NLP models. This is a subset (200 instances) of my data set of 10,000 instances:This the link of the dataset on pastebin
I compare an LSTM model with a glove model and a BERT model. I ...
- 131
0
votes
0
answers
14
views
High-Cardinality Categorical Feature with frequency score
I'm working with a really high-cardinality feature as one of the inputs to my model and I'm using hash-encoded feature embedding rather than one-hot encoding. However, this method is ignoring the ...
- 101
0
votes
1
answer
109
views
Text segmentation problem
I am new to ML and trying to solve problem of text segmentation.
I have a transcript of news show and I want to split this transcript into parts by topic. I tried to google and asked chatgpt and found ...
0
votes
0
answers
15
views
Getting synonyms for a word based on context
I want to get synonyms for a word based on it's use in sentence. for example in the sentence I will book the hotel, book is ...
- 101
0
votes
1
answer
288
views
Deduplication using NLP
I have a product catalog.
The user can add a new product to the catalog. The user can enter some attributes (such as color, weight, etc.) in the text boxes. The user can also mention the description ...
- 391
1
vote
0
answers
111
views
Otimization of similarity search for multiple embeddings by creating a weighted artificial embedding
I have embeddings of text created with a BERT model. A group of these embeddings should be used to find similar embeddings corresponding to this group. I know that you can use average or max (or ...
- 113
3
votes
2
answers
817
views
Semantic search - combine text and image embedding
I have a question regarding combining text and image embeddings for semantic search. The use case is product search on a (B2B) marketplace, we have image(s) and title&description of the products. ...
- 31
0
votes
0
answers
111
views
How can I generate embeddings using previously generated BERT embeddings and feed them to an RNN?
I'm using an unlabeled news corpus to fine-tune a multi-lingual BERT model. After that I'm using those embeddings to generate embeddings for words present in a new labeled dataset. These new ...
- 101
0
votes
1
answer
25
views
Why do varied delimiters on text inputs help training stability?
In the preprint paper Text and code embeddings by contrastive pre-training, the authors describe a Transformer encoder which
maps the input, x and y, to embeddings, vx and vy respectively and the ...
0
votes
0
answers
30
views
Max word embedding
I'm trying to understand the concept behind Max Word / Sentence embeddings also in contrast to mean word embedding. They used it for example in this article. So far I didn't find good resources which ...
- 113
0
votes
1
answer
91
views
How does softmax work for vectors?
In skipgram we predict the context words. That is the output layer before applying the softmax function is a number $V$ of words, where $V$ is the dictionary size. But each word is represented as a ...
- 1
0
votes
0
answers
34
views
Generate vector database for userdata
I need a point in the right direction for the problem I'm trying to solve:
I have a lot of already classified short articles. The articles themselves or a reference to them should be stored in some ...
- 123
0
votes
0
answers
85
views
Clustering with BERT. Why are my clusters overlapped? How to improve BERT embeddings?
I am trying to create BERT embeddings of text data, then use dimensionality reduction and cluster. I tried with some big datasets like amazon reviews and 20newsgroups, but whenever I created ...
0
votes
0
answers
14
views
Align sub-sentences with sentences using embeddings
I would like ask for ideas
So I have sub-sentences, embedded in the context of their respective full-sentences. Then, I have other full-sentences and I would like to find
a) if they have similar sub-...
- 123
0
votes
0
answers
18
views
Creating Word Embeddings using BERT for Machine-Generated Text Data
I have a dataset of machine-generated sequences that are not natural language, but the order of the words in the sequence is important. I want to create word embeddings using BERT to capture the ...
- 101
0
votes
0
answers
78
views
Better results in Document similarity using Word2Vec
I try to cluster similar support-tickets in a technical domain. The support tickets are very domain-specific and are written in various styles, lengths, using abbreviation, etc.
I made a training-...
- 1
1
vote
1
answer
63
views
What should the numerical values of the <startofsentence> and <endofsentence> token vectors be?
I'm trying to build GPT2 from scratch. I understand how to convert each word in a sentence to its respective token index and each token is then converted to its respective word embedding vector. I ...
0
votes
1
answer
134
views
Doubt in ELMO, BERT, Word2Vec
I read an answer on Quora where a NLP Practioner stated that using ELMO and BERT embeddings as input to LSTM or some RNN will defeat the purpose of ELMo and BERT. I am not sure I agree with the above ...
0
votes
0
answers
13
views
Can different corpora sizes in word embedding models over time effect the results? If so, can bootstrapped sampling avoid this?
I have a word embedding model with glove algorithm where I compare association of word X with Y over time, using fourr discreet periods. However, size of my data varies for each period. In Period one ...
2
votes
1
answer
1k
views
Can I use LLM to explain codebase?
I am a Data Engineer, and I am currently assigned a task to refactor an outdated code and rectify any bugs present. However, I am unable to comprehend the code written in the existing codebase. ...
- 158
3
votes
0
answers
358
views
Text embeddings for words or very short sentences with a LLM
I tried to compute the semantic similarity between words or short sentences.
Ex : inflation vs price raising
I have tried the openai embeddings API and cosine distance but the results are very poor.
I ...
- 31
0
votes
0
answers
10
views
Glove vector representation formula derivation - unsymmetric argument
On page 4 of paper "GloVe: Global Vectors for Word Representation".
The author said "Our final model should be invariant under this relabeling, but Eqn. (3) is not." My quesiton is ...
1
vote
1
answer
41
views
Why the label is not explicitly involved in the loss function of skip-gram?
I am recently learning word embedding myself. When learning skip-gram from the paper https://arxiv.org/pdf/1310.4546.pdf[Distributed Representations of Words and Phrases and their Compositionality], I ...
- 11
0
votes
0
answers
24
views
For Q&A NLP system, how to extract the most relevant embedding if it is a combination of top K embeddings?
From my understanding, a typical "AI" Q&A system has a (vector) database of embedded text (from a set of documents). And when a user asks a question, the user's question is embedded and ...
- 1
0
votes
0
answers
20
views
Algorithm of lda2vec in NLP
I was going through lda2vec and was confused on some of the concepts.It is a combination of LDA and word2vec.Word2vec is used to learn dense word vectors and LDA is used to learn the probability ...
- 1
0
votes
0
answers
8
views
Given a Query set, find elements in a pool that are similar to the elements in the query set
I had some curiosity and I was wondering if anyone could shed some light on this. So, let's say I have a query set $Q$ that consists of the following sets of words: {...
- 21