Questions tagged [word-embeddings]

Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.

Filter by
Sorted by
Tagged with
0 votes
0 answers
22 views

"text" parameter in pinecone call from langchain

In this tutorial, I do not understand what "text" refers to vectorstore = Pinecone(index, embeddings.embed_query, "text") Could you please help?...
Karl 17302's user avatar
0 votes
1 answer
30 views

Some fundamental questions about Transformer

In the Transformer framework, a token as an input (time = $t$) $y^t$ is given by a sum of the original embedding of the token $x^t$ plus, a position embedding factor $v^t$, i.e., \begin{align} y^t = x^...
Keyflux's user avatar
  • 101
0 votes
1 answer
17 views

Appropriate input size for nn.Embedding

I’m quite new to using Pytorch and deep learning. What size of unique categories of a categorical variable is appropriate for applying the nn.Embedding ideally (best practices)? for example, if a ...
Любовь Пономарева's user avatar
0 votes
0 answers
14 views

Sum of vector sentence embeddings vs. paragraph embedding

I have been experimenting with the all-MiniLM-L6-v2 model for computing 384-dimensional vector embeddings for text paragraphs. The following code compares the embedding computed for a paragraph with ...
AlwaysLearning's user avatar
0 votes
1 answer
86 views

Training embeddings on own dataset

In my project I follow the retrieval augmented generation (RAG) approach. I want to create embeddings for my own dataset and use it in combination with llama-2. In the dataset are german annual ...
Christian01's user avatar
3 votes
1 answer
46 views

Why do semantically different words produce similar embeddings?

I am comparing words in HuggingFace web UI using e5-small-v2, one of the best vector embedding models: Assuming that the scores are in the range from 0 to 1, how ...
AlwaysLearning's user avatar
2 votes
2 answers
105 views

Why cant we use normalise position encodings instead of the cos and sine encodings used in the Transformer paper?

I'm working with Transformer models for sequence-to-sequence tasks and I'm trying to fully understand the use of positional encodings in these models. In the original "Attention is All You Need&...
mutli-arm-bandit's user avatar
0 votes
2 answers
110 views

Recommended way to embed a text thousands of tokens long?

I've split the text up sections each 512 tokens long and created embeddings for each of them. I want to combine them into 1 embedding for the full text. How do I do that? Is this even recommended? ...
codeananda's user avatar
0 votes
2 answers
27 views

building embeddings for Phrases from scratch

I have a datadet with many phrases which I would like to embed them from scratch. I dont want the cosine of the words in order to get a phrase embedding, this is because the phrases may appear in a ...
Christina Valavani's user avatar
0 votes
0 answers
24 views

Imputing Missing Text Categories in Python Using Word Embeddings/Machine Learning/LLM

I'm working on a dataset where each row represents an entity with several attributes. The dataset includes fields such as ...
nehiljain's user avatar
  • 101
1 vote
1 answer
18 views

word embeddings, what are contextual word embeddings

I m a researcher, a computational linguist and I m working on word embeddings, trying to understand them. What are contextual embeddings and how could I start working and building on my own? Thank you ...
Christina Valavani's user avatar
0 votes
1 answer
30 views

Word embeddings

I m looking into word embedding and I would like to ask if I could train words or sentences in two layers. And if I wanted that one layer is more important, how could I calculate it? For example ...
Christina Valavani's user avatar
1 vote
1 answer
26 views

ELMO embeddings

Could somebody tell me how does elmo work? Is it good for phrase embedding too? I m looking for phrase embeddings. Thank You in advance.
Christina Valavani's user avatar
0 votes
1 answer
22 views

more insights about Word2Vec implementation

As we know Word2Vec is non-contextual embedding (at word level). As per my knowledge, BOW is statistical embedding technique (word level). we can perform Word2Vec embedding in two approaches: 1. CBOW. ...
tovijayak's user avatar
0 votes
1 answer
47 views

What do averaged word vectors represent?

Assume you have high-dimensional word embeddings (d > 100) for a large number of words (|V| > 100,000) calculated over a huge non-specialized natural language corpus. Assume you have taken the ...
Hans-Peter Stricker's user avatar
0 votes
1 answer
25 views

How do we get output layer in skip-gram?

Could you please explain how do we get output layer in this architecture (vectors [0.2, 0.8, -1.4, 1.2] and ...
manabou11's user avatar
  • 101
1 vote
0 answers
42 views

I can't get good performance from BERT

I trained NLP models. This is a subset (200 instances) of my data set of 10,000 instances:This the link of the dataset on pastebin I compare an LSTM model with a glove model and a BERT model. I ...
Seydou GORO's user avatar
0 votes
0 answers
12 views

High-Cardinality Categorical Feature with frequency score

I'm working with a really high-cardinality feature as one of the inputs to my model and I'm using hash-encoded feature embedding rather than one-hot encoding. However, this method is ignoring the ...
UK-07's user avatar
  • 101
0 votes
1 answer
61 views

Text segmentation problem

I am new to ML and trying to solve problem of text segmentation. I have a transcript of news show and I want to split this transcript into parts by topic. I tried to google and asked chatgpt and found ...
Oleg Bovykin's user avatar
0 votes
0 answers
14 views

Getting synonyms for a word based on context

I want to get synonyms for a word based on it's use in sentence. for example in the sentence I will book the hotel, book is ...
Pooya Estakhri's user avatar
0 votes
1 answer
192 views

Deduplication using NLP

I have a product catalog. The user can add a new product to the catalog. The user can enter some attributes (such as color, weight, etc.) in the text boxes. The user can also mention the description ...
Shrinidhi M's user avatar
1 vote
0 answers
85 views

Otimization of similarity search for multiple embeddings by creating a weighted artificial embedding

I have embeddings of text created with a BERT model. A group of these embeddings should be used to find similar embeddings corresponding to this group. I know that you can use average or max (or ...
soph's user avatar
  • 113
3 votes
2 answers
506 views

Semantic search - combine text and image embedding

I have a question regarding combining text and image embeddings for semantic search. The use case is product search on a (B2B) marketplace, we have image(s) and title&description of the products. ...
Steven's user avatar
  • 31
0 votes
0 answers
97 views

How can I generate embeddings using previously generated BERT embeddings and feed them to an RNN?

I'm using an unlabeled news corpus to fine-tune a multi-lingual BERT model. After that I'm using those embeddings to generate embeddings for words present in a new labeled dataset. These new ...
Debbie's user avatar
  • 101
0 votes
1 answer
25 views

Why do varied delimiters on text inputs help training stability?

In the preprint paper Text and code embeddings by contrastive pre-training, the authors describe a Transformer encoder which maps the input, x and y, to embeddings, vx and vy respectively and the ...
Robbie Palmer's user avatar
0 votes
0 answers
21 views

Max word embedding

I'm trying to understand the concept behind Max Word / Sentence embeddings also in contrast to mean word embedding. They used it for example in this article. So far I didn't find good resources which ...
soph's user avatar
  • 113
0 votes
1 answer
69 views

How does softmax work for vectors?

In skipgram we predict the context words. That is the output layer before applying the softmax function is a number $V$ of words, where $V$ is the dictionary size. But each word is represented as a ...
Ruediger's user avatar
0 votes
0 answers
23 views

Generate vector database for userdata

I need a point in the right direction for the problem I'm trying to solve: I have a lot of already classified short articles. The articles themselves or a reference to them should be stored in some ...
mathi1651 's user avatar
0 votes
0 answers
54 views

Clustering with BERT. Why are my clusters overlapped? How to improve BERT embeddings?

I am trying to create BERT embeddings of text data, then use dimensionality reduction and cluster. I tried with some big datasets like amazon reviews and 20newsgroups, but whenever I created ...
William Smith's user avatar
0 votes
0 answers
7 views

Align sub-sentences with sentences using embeddings

I would like ask for ideas So I have sub-sentences, embedded in the context of their respective full-sentences. Then, I have other full-sentences and I would like to find a) if they have similar sub-...
aqua's user avatar
  • 123
0 votes
0 answers
17 views

Creating Word Embeddings using BERT for Machine-Generated Text Data

I have a dataset of machine-generated sequences that are not natural language, but the order of the words in the sequence is important. I want to create word embeddings using BERT to capture the ...
Bella_18's user avatar
  • 101
0 votes
0 answers
60 views

Better results in Document similarity using Word2Vec

I try to cluster similar support-tickets in a technical domain. The support tickets are very domain-specific and are written in various styles, lengths, using abbreviation, etc. I made a training-...
Roland's user avatar
  • 1
1 vote
1 answer
36 views

What should the numerical values of the <startofsentence> and <endofsentence> token vectors be?

I'm trying to build GPT2 from scratch. I understand how to convert each word in a sentence to its respective token index and each token is then converted to its respective word embedding vector. I ...
Austin Capobianco's user avatar
0 votes
1 answer
93 views

Doubt in ELMO, BERT, Word2Vec

I read an answer on Quora where a NLP Practioner stated that using ELMO and BERT embeddings as input to LSTM or some RNN will defeat the purpose of ELMo and BERT. I am not sure I agree with the above ...
NeverGiveUp's user avatar
0 votes
0 answers
13 views

Can different corpora sizes in word embedding models over time effect the results? If so, can bootstrapped sampling avoid this?

I have a word embedding model with glove algorithm where I compare association of word X with Y over time, using fourr discreet periods. However, size of my data varies for each period. In Period one ...
Iamembarassed123's user avatar
2 votes
1 answer
1k views

Can I use LLM to explain codebase?

I am a Data Engineer, and I am currently assigned a task to refactor an outdated code and rectify any bugs present. However, I am unable to comprehend the code written in the existing codebase. ...
Yiffany's user avatar
  • 158
2 votes
0 answers
282 views

Text embeddings for words or very short sentences with a LLM

I tried to compute the semantic similarity between words or short sentences. Ex : inflation vs price raising I have tried the openai embeddings API and cosine distance but the results are very poor. I ...
user2479920's user avatar
0 votes
0 answers
10 views

Glove vector representation formula derivation - unsymmetric argument

On page 4 of paper "GloVe: Global Vectors for Word Representation". The author said "Our final model should be invariant under this relabeling, but Eqn. (3) is not." My quesiton is ...
Sichao Young's user avatar
1 vote
1 answer
32 views

Why the label is not explicitly involved in the loss function of skip-gram?

I am recently learning word embedding myself. When learning skip-gram from the paper https://arxiv.org/pdf/1310.4546.pdf[Distributed Representations of Words and Phrases and their Compositionality], I ...
JQ_SE's user avatar
  • 11
0 votes
0 answers
20 views

For Q&A NLP system, how to extract the most relevant embedding if it is a combination of top K embeddings?

From my understanding, a typical "AI" Q&A system has a (vector) database of embedded text (from a set of documents). And when a user asks a question, the user's question is embedded and ...
siddgood's user avatar
0 votes
0 answers
19 views

Algorithm of lda2vec in NLP

I was going through lda2vec and was confused on some of the concepts.It is a combination of LDA and word2vec.Word2vec is used to learn dense word vectors and LDA is used to learn the probability ...
Alex's user avatar
  • 1
0 votes
0 answers
8 views

Given a Query set, find elements in a pool that are similar to the elements in the query set

I had some curiosity and I was wondering if anyone could shed some light on this. So, let's say I have a query set $Q$ that consists of the following sets of words: {...
M. Fire's user avatar
  • 21
0 votes
0 answers
29 views

Multimodal semantic search

Context: I am interested in using the potential of using embeddings to consolidate texts/documents with very different surface forms into one searchable database - in other words, to produce a sort of ...
Greggs's user avatar
  • 121
0 votes
0 answers
25 views

Understanding gradient of skip gram

I am trying to understand gradient calculation for skip gram with softmax output and cross entropy loss. I am referring these articles: 1, 2, 3. The all calculate the error as follows: $$E=-\sum_{c=1}...
RajS's user avatar
  • 103
3 votes
1 answer
32 views

Is there a reference dataset for contextual similarity?

I'm doing some experiments with word embeddings to try to capture context-aware similarity, so that for example the word pair apple - hardware, are very dissimilar in the context of a fruit store, but ...
Jorgemar's user avatar
  • 241
0 votes
0 answers
8 views

Optimize wordembedding and neural network at the same time

I have a lot of (domain)-specific text that I want to classify into 100+ categories. I want to train a wordembedding (FastText) and use that in conjuction with a CNN, thus I'm running into the problem ...
CutePoison's user avatar
0 votes
0 answers
237 views

Can BM25 be used as an embedding algorithm?

I'v studied about BM25 algorithm. Untill now, I couldn't find an implementation of BM25 to give me an embedding of a text like TfidfTransformer and ...
Mohsen Mahmoodzadeh's user avatar
0 votes
0 answers
132 views

Finetuning fasttext with unlabeled text corpus

I am training a classifier which is supposed to take the name of a product as input. For this purpose I want to finetune a pre-existing fasttext model on my article names. My code looks like this <...
christallclear's user avatar
1 vote
1 answer
42 views

Understand the interpretability of word embeddings

When reading the Tensorflow tutorial for Word Embeddings, I found two notes that confuse me: Note: Experimentally, you may be able to produce more interpretable embeddings by using a simpler model. ...
Tran Khanh's user avatar
2 votes
1 answer
817 views

How do we evaluate the outputs of text generation models?

Evaluation of a wide variety of natural language generation (NLG) tasks is difficult. For instance, for a question answering model, it is hard for a human to quantify how well the model has answered a ...
Greggs's user avatar
  • 121

1
2 3 4 5
11