Questions tagged [word-embeddings]

Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.

Filter by
Sorted by
Tagged with
1 vote
1 answer
48 views

How does Bert masked language modelling task make sense if half the time the next sentence is wrong context in the sequence passed through the encoder

Bert has two types of tasks that it uses to learn contextual word embeddings: Masked word prediction Next sentence prediction I have read the paper and even there the training details are a little ...
figs_and_nuts's user avatar
0 votes
0 answers
31 views

How was the word2vec model trained?

Let's take the CBOW (continuous bag of words) model as the example. Suppose that, there are $c$ context words, each of which is a one-hot encoding vector. So the total number of elements of input ...
J. Doe's user avatar
  • 46
0 votes
0 answers
10 views

For a fine-tuning a transformer to type like a specific person, should I use sentence semantic embeddings or word semantic embeddings

I'm not clear on the pros and cons of each one for this particular task. Is there even a meaningful difference? My guess is using semantic embeddings for words will be better in nearly all cases ...
Austin Capobianco's user avatar
0 votes
1 answer
30 views

What does maximize average log probability mean?

In the word2vec paper (https://arxiv.org/pdf/1310.4546.pdf) that introduces the skip-gram algorithm we encounter this phrase: which says that we maximize the average log probability. Can someone help ...
Claudiu Creanga's user avatar
0 votes
0 answers
24 views

Is openAI text generation models an extension of embedding models?

we can creating embeddings using below code ...
Vinay Sharma's user avatar
1 vote
1 answer
56 views

A question about contextual embeddings in the decoder only transformer architecture (gpt)

I am reading up on the decoder only architecture Relevant excerpts: We can use any model that maps token sequences into contextual embeddings (e.g., LSTMs, Transformers): $$\phi : V^L \to R^{d \times ...
figs_and_nuts's user avatar
0 votes
1 answer
25 views

How to use location information as feature?

I have a location feature in a dataset. Some examples are: London, Uk; Sheefield Town, Ohio; UK ; North Carolina. etc. How to encode them into features? Is there any word embeddings suitable for such ...
Alex_ban's user avatar
  • 101
0 votes
0 answers
204 views

How to get Llama-2 Rotary Embeddings?

I want to get the Llama-2 rotary embeddings. I do print(model) and get the following output: In the picture I highlight the rotary embeddings. How can get the ...
Christian01's user avatar
0 votes
0 answers
27 views

"text" parameter in pinecone call from langchain

In this tutorial, I do not understand what "text" refers to vectorstore = Pinecone(index, embeddings.embed_query, "text") Could you please help?...
Karl 17302's user avatar
0 votes
1 answer
41 views

Some fundamental questions about Transformer

In the Transformer framework, a token as an input (time = $t$) $y^t$ is given by a sum of the original embedding of the token $x^t$ plus, a position embedding factor $v^t$, i.e., \begin{align} y^t = x^...
Keyflux's user avatar
  • 101
0 votes
1 answer
28 views

Appropriate input size for nn.Embedding

I’m quite new to using Pytorch and deep learning. What size of unique categories of a categorical variable is appropriate for applying the nn.Embedding ideally (best practices)? for example, if a ...
Любовь Пономарева's user avatar
0 votes
0 answers
60 views

Sum of vector sentence embeddings vs. paragraph embedding

I have been experimenting with the all-MiniLM-L6-v2 model for computing 384-dimensional vector embeddings for text paragraphs. The following code compares the embedding computed for a paragraph with ...
AlwaysLearning's user avatar
0 votes
1 answer
381 views

Training embeddings on own dataset

In my project I follow the retrieval augmented generation (RAG) approach. I want to create embeddings for my own dataset and use it in combination with llama-2. In the dataset are german annual ...
Christian01's user avatar
3 votes
1 answer
80 views

Why do semantically different words produce similar embeddings?

I am comparing words in HuggingFace web UI using e5-small-v2, one of the best vector embedding models: Assuming that the scores are in the range from 0 to 1, how ...
AlwaysLearning's user avatar
2 votes
2 answers
268 views

Why cant we use normalise position encodings instead of the cos and sine encodings used in the Transformer paper?

I'm working with Transformer models for sequence-to-sequence tasks and I'm trying to fully understand the use of positional encodings in these models. In the original "Attention is All You Need&...
mutli-arm-bandit's user avatar
0 votes
2 answers
531 views

Recommended way to embed a text thousands of tokens long?

I've split the text up sections each 512 tokens long and created embeddings for each of them. I want to combine them into 1 embedding for the full text. How do I do that? Is this even recommended? ...
codeananda's user avatar
0 votes
2 answers
32 views

building embeddings for Phrases from scratch

I have a datadet with many phrases which I would like to embed them from scratch. I dont want the cosine of the words in order to get a phrase embedding, this is because the phrases may appear in a ...
Christina Valavani's user avatar
0 votes
0 answers
30 views

Imputing Missing Text Categories in Python Using Word Embeddings/Machine Learning/LLM

I'm working on a dataset where each row represents an entity with several attributes. The dataset includes fields such as ...
nehiljain's user avatar
  • 101
1 vote
1 answer
25 views

word embeddings, what are contextual word embeddings

I m a researcher, a computational linguist and I m working on word embeddings, trying to understand them. What are contextual embeddings and how could I start working and building on my own? Thank you ...
Christina Valavani's user avatar
0 votes
1 answer
38 views

Word embeddings

I m looking into word embedding and I would like to ask if I could train words or sentences in two layers. And if I wanted that one layer is more important, how could I calculate it? For example ...
Christina Valavani's user avatar
1 vote
1 answer
28 views

ELMO embeddings

Could somebody tell me how does elmo work? Is it good for phrase embedding too? I m looking for phrase embeddings. Thank You in advance.
Christina Valavani's user avatar
0 votes
1 answer
23 views

more insights about Word2Vec implementation

As we know Word2Vec is non-contextual embedding (at word level). As per my knowledge, BOW is statistical embedding technique (word level). we can perform Word2Vec embedding in two approaches: 1. CBOW. ...
tovijayak's user avatar
0 votes
1 answer
49 views

What do averaged word vectors represent?

Assume you have high-dimensional word embeddings (d > 100) for a large number of words (|V| > 100,000) calculated over a huge non-specialized natural language corpus. Assume you have taken the ...
Hans-Peter Stricker's user avatar
0 votes
1 answer
36 views

How do we get output layer in skip-gram?

Could you please explain how do we get output layer in this architecture (vectors [0.2, 0.8, -1.4, 1.2] and ...
manabou11's user avatar
  • 101
1 vote
0 answers
47 views

I can't get good performance from BERT

I trained NLP models. This is a subset (200 instances) of my data set of 10,000 instances:This the link of the dataset on pastebin I compare an LSTM model with a glove model and a BERT model. I ...
Seydou GORO's user avatar
0 votes
0 answers
14 views

High-Cardinality Categorical Feature with frequency score

I'm working with a really high-cardinality feature as one of the inputs to my model and I'm using hash-encoded feature embedding rather than one-hot encoding. However, this method is ignoring the ...
UK-07's user avatar
  • 101
0 votes
1 answer
109 views

Text segmentation problem

I am new to ML and trying to solve problem of text segmentation. I have a transcript of news show and I want to split this transcript into parts by topic. I tried to google and asked chatgpt and found ...
Oleg Bovykin's user avatar
0 votes
0 answers
15 views

Getting synonyms for a word based on context

I want to get synonyms for a word based on it's use in sentence. for example in the sentence I will book the hotel, book is ...
Pooya Estakhri's user avatar
0 votes
1 answer
288 views

Deduplication using NLP

I have a product catalog. The user can add a new product to the catalog. The user can enter some attributes (such as color, weight, etc.) in the text boxes. The user can also mention the description ...
Shrinidhi M's user avatar
1 vote
0 answers
111 views

Otimization of similarity search for multiple embeddings by creating a weighted artificial embedding

I have embeddings of text created with a BERT model. A group of these embeddings should be used to find similar embeddings corresponding to this group. I know that you can use average or max (or ...
soph's user avatar
  • 113
3 votes
2 answers
817 views

Semantic search - combine text and image embedding

I have a question regarding combining text and image embeddings for semantic search. The use case is product search on a (B2B) marketplace, we have image(s) and title&description of the products. ...
Steven's user avatar
  • 31
0 votes
0 answers
111 views

How can I generate embeddings using previously generated BERT embeddings and feed them to an RNN?

I'm using an unlabeled news corpus to fine-tune a multi-lingual BERT model. After that I'm using those embeddings to generate embeddings for words present in a new labeled dataset. These new ...
Debbie's user avatar
  • 101
0 votes
1 answer
25 views

Why do varied delimiters on text inputs help training stability?

In the preprint paper Text and code embeddings by contrastive pre-training, the authors describe a Transformer encoder which maps the input, x and y, to embeddings, vx and vy respectively and the ...
Robbie Palmer's user avatar
0 votes
0 answers
30 views

Max word embedding

I'm trying to understand the concept behind Max Word / Sentence embeddings also in contrast to mean word embedding. They used it for example in this article. So far I didn't find good resources which ...
soph's user avatar
  • 113
0 votes
1 answer
91 views

How does softmax work for vectors?

In skipgram we predict the context words. That is the output layer before applying the softmax function is a number $V$ of words, where $V$ is the dictionary size. But each word is represented as a ...
Ruediger's user avatar
0 votes
0 answers
34 views

Generate vector database for userdata

I need a point in the right direction for the problem I'm trying to solve: I have a lot of already classified short articles. The articles themselves or a reference to them should be stored in some ...
mathi1651 's user avatar
0 votes
0 answers
85 views

Clustering with BERT. Why are my clusters overlapped? How to improve BERT embeddings?

I am trying to create BERT embeddings of text data, then use dimensionality reduction and cluster. I tried with some big datasets like amazon reviews and 20newsgroups, but whenever I created ...
William Smith's user avatar
0 votes
0 answers
14 views

Align sub-sentences with sentences using embeddings

I would like ask for ideas So I have sub-sentences, embedded in the context of their respective full-sentences. Then, I have other full-sentences and I would like to find a) if they have similar sub-...
aqua's user avatar
  • 123
0 votes
0 answers
18 views

Creating Word Embeddings using BERT for Machine-Generated Text Data

I have a dataset of machine-generated sequences that are not natural language, but the order of the words in the sequence is important. I want to create word embeddings using BERT to capture the ...
Bella_18's user avatar
  • 101
0 votes
0 answers
78 views

Better results in Document similarity using Word2Vec

I try to cluster similar support-tickets in a technical domain. The support tickets are very domain-specific and are written in various styles, lengths, using abbreviation, etc. I made a training-...
Roland's user avatar
  • 1
1 vote
1 answer
63 views

What should the numerical values of the <startofsentence> and <endofsentence> token vectors be?

I'm trying to build GPT2 from scratch. I understand how to convert each word in a sentence to its respective token index and each token is then converted to its respective word embedding vector. I ...
Austin Capobianco's user avatar
0 votes
1 answer
134 views

Doubt in ELMO, BERT, Word2Vec

I read an answer on Quora where a NLP Practioner stated that using ELMO and BERT embeddings as input to LSTM or some RNN will defeat the purpose of ELMo and BERT. I am not sure I agree with the above ...
NeverGiveUp's user avatar
0 votes
0 answers
13 views

Can different corpora sizes in word embedding models over time effect the results? If so, can bootstrapped sampling avoid this?

I have a word embedding model with glove algorithm where I compare association of word X with Y over time, using fourr discreet periods. However, size of my data varies for each period. In Period one ...
Iamembarassed123's user avatar
2 votes
1 answer
1k views

Can I use LLM to explain codebase?

I am a Data Engineer, and I am currently assigned a task to refactor an outdated code and rectify any bugs present. However, I am unable to comprehend the code written in the existing codebase. ...
Yiffany's user avatar
  • 158
3 votes
0 answers
358 views

Text embeddings for words or very short sentences with a LLM

I tried to compute the semantic similarity between words or short sentences. Ex : inflation vs price raising I have tried the openai embeddings API and cosine distance but the results are very poor. I ...
user2479920's user avatar
0 votes
0 answers
10 views

Glove vector representation formula derivation - unsymmetric argument

On page 4 of paper "GloVe: Global Vectors for Word Representation". The author said "Our final model should be invariant under this relabeling, but Eqn. (3) is not." My quesiton is ...
Sichao Young's user avatar
1 vote
1 answer
41 views

Why the label is not explicitly involved in the loss function of skip-gram?

I am recently learning word embedding myself. When learning skip-gram from the paper https://arxiv.org/pdf/1310.4546.pdf[Distributed Representations of Words and Phrases and their Compositionality], I ...
JQ_SE's user avatar
  • 11
0 votes
0 answers
24 views

For Q&A NLP system, how to extract the most relevant embedding if it is a combination of top K embeddings?

From my understanding, a typical "AI" Q&A system has a (vector) database of embedded text (from a set of documents). And when a user asks a question, the user's question is embedded and ...
siddgood's user avatar
0 votes
0 answers
20 views

Algorithm of lda2vec in NLP

I was going through lda2vec and was confused on some of the concepts.It is a combination of LDA and word2vec.Word2vec is used to learn dense word vectors and LDA is used to learn the probability ...
Alex's user avatar
  • 1
0 votes
0 answers
8 views

Given a Query set, find elements in a pool that are similar to the elements in the query set

I had some curiosity and I was wondering if anyone could shed some light on this. So, let's say I have a query set $Q$ that consists of the following sets of words: {...
M. Fire's user avatar
  • 21

1
2 3 4 5
11