Questions tagged [word-embeddings]
Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.
502
questions
0
votes
0
answers
22
views
"text" parameter in pinecone call from langchain
In this tutorial, I do not understand what "text" refers to
vectorstore = Pinecone(index, embeddings.embed_query, "text")
Could you please help?...
0
votes
1
answer
30
views
Some fundamental questions about Transformer
In the Transformer framework, a token as an input (time = $t$) $y^t$ is given by a sum of the original embedding of the token $x^t$ plus, a position embedding factor $v^t$, i.e.,
\begin{align}
y^t = x^...
- 101
0
votes
1
answer
17
views
Appropriate input size for nn.Embedding
I’m quite new to using Pytorch and deep learning. What size of unique categories of a categorical variable is appropriate for applying the nn.Embedding ideally (best practices)? for example, if a ...
0
votes
0
answers
14
views
Sum of vector sentence embeddings vs. paragraph embedding
I have been experimenting with the all-MiniLM-L6-v2 model for computing 384-dimensional vector embeddings for text paragraphs. The following code compares the embedding computed for a paragraph with ...
- 141
0
votes
1
answer
86
views
Training embeddings on own dataset
In my project I follow the retrieval augmented generation (RAG) approach.
I want to create embeddings for my own dataset and use it in combination with llama-2.
In the dataset are german annual ...
- 101
3
votes
1
answer
46
views
Why do semantically different words produce similar embeddings?
I am comparing words in HuggingFace web UI using e5-small-v2, one of the best vector embedding models:
Assuming that the scores are in the range from 0 to 1, how ...
- 141
2
votes
2
answers
105
views
Why cant we use normalise position encodings instead of the cos and sine encodings used in the Transformer paper?
I'm working with Transformer models for sequence-to-sequence tasks and I'm trying to fully understand the use of positional encodings in these models.
In the original "Attention is All You Need&...
0
votes
2
answers
110
views
Recommended way to embed a text thousands of tokens long?
I've split the text up sections each 512 tokens long and created embeddings for each of them.
I want to combine them into 1 embedding for the full text. How do I do that? Is this even recommended?
...
- 268
0
votes
2
answers
27
views
building embeddings for Phrases from scratch
I have a datadet with many phrases which I would like to embed them from scratch. I dont want the cosine of the words in order to get a phrase embedding, this is because the phrases may appear in a ...
0
votes
0
answers
24
views
Imputing Missing Text Categories in Python Using Word Embeddings/Machine Learning/LLM
I'm working on a dataset where each row represents an entity with several attributes. The dataset includes fields such as ...
- 101
1
vote
1
answer
18
views
word embeddings, what are contextual word embeddings
I m a researcher, a computational linguist and I m working on word embeddings, trying to understand them.
What are contextual embeddings and how could I start working and building on my own?
Thank you ...
0
votes
1
answer
30
views
Word embeddings
I m looking into word embedding and I would like to ask if I could train words or sentences in two layers. And if I wanted that one layer is more important, how could I calculate it? For example ...
1
vote
1
answer
26
views
ELMO embeddings
Could somebody tell me how does elmo work?
Is it good for phrase embedding too?
I m looking for phrase embeddings.
Thank You in advance.
0
votes
1
answer
22
views
more insights about Word2Vec implementation
As we know Word2Vec is non-contextual embedding (at word level). As per my knowledge, BOW is statistical embedding technique (word level).
we can perform Word2Vec embedding in two approaches: 1. CBOW. ...
- 37
0
votes
1
answer
47
views
What do averaged word vectors represent?
Assume you have high-dimensional word embeddings (d > 100) for a large number of words (|V| > 100,000) calculated over a huge non-specialized natural language corpus. Assume you have taken the ...
0
votes
1
answer
25
views
How do we get output layer in skip-gram?
Could you please explain how do we get output layer in this architecture (vectors [0.2, 0.8, -1.4, 1.2] and ...
- 101
1
vote
0
answers
42
views
I can't get good performance from BERT
I trained NLP models. This is a subset (200 instances) of my data set of 10,000 instances:This the link of the dataset on pastebin
I compare an LSTM model with a glove model and a BERT model. I ...
- 131
0
votes
0
answers
12
views
High-Cardinality Categorical Feature with frequency score
I'm working with a really high-cardinality feature as one of the inputs to my model and I'm using hash-encoded feature embedding rather than one-hot encoding. However, this method is ignoring the ...
- 101
0
votes
1
answer
61
views
Text segmentation problem
I am new to ML and trying to solve problem of text segmentation.
I have a transcript of news show and I want to split this transcript into parts by topic. I tried to google and asked chatgpt and found ...
0
votes
0
answers
14
views
Getting synonyms for a word based on context
I want to get synonyms for a word based on it's use in sentence. for example in the sentence I will book the hotel, book is ...
- 101
0
votes
1
answer
192
views
Deduplication using NLP
I have a product catalog.
The user can add a new product to the catalog. The user can enter some attributes (such as color, weight, etc.) in the text boxes. The user can also mention the description ...
- 381
1
vote
0
answers
85
views
Otimization of similarity search for multiple embeddings by creating a weighted artificial embedding
I have embeddings of text created with a BERT model. A group of these embeddings should be used to find similar embeddings corresponding to this group. I know that you can use average or max (or ...
- 113
3
votes
2
answers
506
views
Semantic search - combine text and image embedding
I have a question regarding combining text and image embeddings for semantic search. The use case is product search on a (B2B) marketplace, we have image(s) and title&description of the products. ...
- 31
0
votes
0
answers
97
views
How can I generate embeddings using previously generated BERT embeddings and feed them to an RNN?
I'm using an unlabeled news corpus to fine-tune a multi-lingual BERT model. After that I'm using those embeddings to generate embeddings for words present in a new labeled dataset. These new ...
- 101
0
votes
1
answer
25
views
Why do varied delimiters on text inputs help training stability?
In the preprint paper Text and code embeddings by contrastive pre-training, the authors describe a Transformer encoder which
maps the input, x and y, to embeddings, vx and vy respectively and the ...
0
votes
0
answers
21
views
Max word embedding
I'm trying to understand the concept behind Max Word / Sentence embeddings also in contrast to mean word embedding. They used it for example in this article. So far I didn't find good resources which ...
- 113
0
votes
1
answer
69
views
How does softmax work for vectors?
In skipgram we predict the context words. That is the output layer before applying the softmax function is a number $V$ of words, where $V$ is the dictionary size. But each word is represented as a ...
- 1
0
votes
0
answers
23
views
Generate vector database for userdata
I need a point in the right direction for the problem I'm trying to solve:
I have a lot of already classified short articles. The articles themselves or a reference to them should be stored in some ...
- 123
0
votes
0
answers
54
views
Clustering with BERT. Why are my clusters overlapped? How to improve BERT embeddings?
I am trying to create BERT embeddings of text data, then use dimensionality reduction and cluster. I tried with some big datasets like amazon reviews and 20newsgroups, but whenever I created ...
0
votes
0
answers
7
views
Align sub-sentences with sentences using embeddings
I would like ask for ideas
So I have sub-sentences, embedded in the context of their respective full-sentences. Then, I have other full-sentences and I would like to find
a) if they have similar sub-...
- 123
0
votes
0
answers
17
views
Creating Word Embeddings using BERT for Machine-Generated Text Data
I have a dataset of machine-generated sequences that are not natural language, but the order of the words in the sequence is important. I want to create word embeddings using BERT to capture the ...
- 101
0
votes
0
answers
60
views
Better results in Document similarity using Word2Vec
I try to cluster similar support-tickets in a technical domain. The support tickets are very domain-specific and are written in various styles, lengths, using abbreviation, etc.
I made a training-...
- 1
1
vote
1
answer
36
views
What should the numerical values of the <startofsentence> and <endofsentence> token vectors be?
I'm trying to build GPT2 from scratch. I understand how to convert each word in a sentence to its respective token index and each token is then converted to its respective word embedding vector. I ...
0
votes
1
answer
93
views
Doubt in ELMO, BERT, Word2Vec
I read an answer on Quora where a NLP Practioner stated that using ELMO and BERT embeddings as input to LSTM or some RNN will defeat the purpose of ELMo and BERT. I am not sure I agree with the above ...
0
votes
0
answers
13
views
Can different corpora sizes in word embedding models over time effect the results? If so, can bootstrapped sampling avoid this?
I have a word embedding model with glove algorithm where I compare association of word X with Y over time, using fourr discreet periods. However, size of my data varies for each period. In Period one ...
2
votes
1
answer
1k
views
Can I use LLM to explain codebase?
I am a Data Engineer, and I am currently assigned a task to refactor an outdated code and rectify any bugs present. However, I am unable to comprehend the code written in the existing codebase. ...
- 158
2
votes
0
answers
282
views
Text embeddings for words or very short sentences with a LLM
I tried to compute the semantic similarity between words or short sentences.
Ex : inflation vs price raising
I have tried the openai embeddings API and cosine distance but the results are very poor.
I ...
- 21
0
votes
0
answers
10
views
Glove vector representation formula derivation - unsymmetric argument
On page 4 of paper "GloVe: Global Vectors for Word Representation".
The author said "Our final model should be invariant under this relabeling, but Eqn. (3) is not." My quesiton is ...
1
vote
1
answer
32
views
Why the label is not explicitly involved in the loss function of skip-gram?
I am recently learning word embedding myself. When learning skip-gram from the paper https://arxiv.org/pdf/1310.4546.pdf[Distributed Representations of Words and Phrases and their Compositionality], I ...
- 11
0
votes
0
answers
20
views
For Q&A NLP system, how to extract the most relevant embedding if it is a combination of top K embeddings?
From my understanding, a typical "AI" Q&A system has a (vector) database of embedded text (from a set of documents). And when a user asks a question, the user's question is embedded and ...
- 1
0
votes
0
answers
19
views
Algorithm of lda2vec in NLP
I was going through lda2vec and was confused on some of the concepts.It is a combination of LDA and word2vec.Word2vec is used to learn dense word vectors and LDA is used to learn the probability ...
- 1
0
votes
0
answers
8
views
Given a Query set, find elements in a pool that are similar to the elements in the query set
I had some curiosity and I was wondering if anyone could shed some light on this. So, let's say I have a query set $Q$ that consists of the following sets of words: {...
- 21
0
votes
0
answers
29
views
Multimodal semantic search
Context: I am interested in using the potential of using embeddings to consolidate texts/documents with very different surface forms into one searchable database - in other words, to produce a sort of ...
- 121
0
votes
0
answers
25
views
Understanding gradient of skip gram
I am trying to understand gradient calculation for skip gram with softmax output and cross entropy loss.
I am referring these articles: 1, 2, 3.
The all calculate the error as follows:
$$E=-\sum_{c=1}...
- 103
3
votes
1
answer
32
views
Is there a reference dataset for contextual similarity?
I'm doing some experiments with word embeddings to try to capture context-aware similarity, so that for example the word pair apple - hardware, are very dissimilar in the context of a fruit store, but ...
- 241
0
votes
0
answers
8
views
Optimize wordembedding and neural network at the same time
I have a lot of (domain)-specific text that I want to classify into 100+ categories.
I want to train a wordembedding (FastText) and use that in conjuction with a CNN, thus I'm running into the problem ...
- 450
0
votes
0
answers
237
views
Can BM25 be used as an embedding algorithm?
I'v studied about BM25 algorithm. Untill now, I couldn't find an implementation of BM25 to give me an embedding of a text like TfidfTransformer and ...
0
votes
0
answers
132
views
Finetuning fasttext with unlabeled text corpus
I am training a classifier which is supposed to take the name of a product as input. For this purpose I want to finetune a pre-existing fasttext model on my article names.
My code looks like this
<...
1
vote
1
answer
42
views
Understand the interpretability of word embeddings
When reading the Tensorflow tutorial for Word Embeddings, I found two notes that confuse me:
Note: Experimentally, you may be able to produce more interpretable embeddings by using a simpler model. ...
- 13
2
votes
1
answer
817
views
How do we evaluate the outputs of text generation models?
Evaluation of a wide variety of natural language generation (NLG) tasks is difficult. For instance, for a question answering model, it is hard for a human to quantify how well the model has answered a ...
- 121