Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [word-embeddings]

Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.

Filter by
Sorted by
Tagged with
1 vote
1 answer
40 views

How OpenAI embeddings work?

I was looking at the Stanford CS224N NLP with Deep Learning lecture, and in the first two videos, we are introduced to word2vec models. The high-level idea mentioned was that we have a 'big corpus' of ...
mw981's user avatar
  • 13
0 votes
0 answers
59 views

"No sentence-transformers model found with name" on huggingface even though it exists

I am trying to use infgrad/stella-base-en-v2 on hugging to generate embeddings using langchain The model exists on the huggingface hub The model is listed on the MTEB leaderboard The model has ...
figs_and_nuts's user avatar
1 vote
0 answers
19 views

word2vec predicts the same word for all inputs

i build word2vec network with 2 linear layers from pytorch. for every word as an input i consistently train model to predict words before and after, for example: i was visiting my grandma's house, for ...
Тима 's user avatar
0 votes
0 answers
21 views

Insights about W0rd2Vec

As per my knowledge, Word2Vec is belongs to non-contextual embedding technique. this have only semantic relationship between words. We can implement Word2Vec, either in CBoW or skip-gram model. but i ...
Tovlk's user avatar
  • 43
0 votes
1 answer
75 views

How to combine two vector embeddings into one?

I want to use OpenCLIP for generating embeddings for each slide in an array of pptx presentations. To improve the quality of the results, I want to vectorize both slide text content and preview images....
Olek Gornostal's user avatar
0 votes
0 answers
24 views

Approach for Multi-class Classification of texts

I'm trying to do a project where I have paragraphs and I need to classify them into multiple labels. The dataset is around 40k rows with labels. I understand there is no one right approach but should ...
Shaurya Uniyal's user avatar
1 vote
1 answer
50 views

How can I use contextual embeddings with BERT for sentiment analysis/classification

I have a BERT model which I want to use for sentiment analysis/classification. E.g. I have some tweets that need to get a POSITIVE,NEGATIVE or NEUTRAL label. I can't understand how contextual ...
average_discrete_math_enjoyer's user avatar
0 votes
0 answers
12 views

Accuracy decreased after using google word2vec model for a sentiment classification [NLP][word-embedding]

I am using Amazon fine food reviews for a sentiment classification project. while I used my dataset corpus to train avg word2vec , I was getting an accuracy of 89 %. by using BOW and TF-IDF, i was ...
Abhishek Kumar Yadav's user avatar
0 votes
0 answers
25 views

What is the state of art production search algorithm right now for semantic search? LSH? or other clustering method?

Trying to implement semantic search for high cardinality embedding for my own learning purpose, so far LSH seems promising, but I am wondering what is the state of algorithm big tech company are using ...
progr's user avatar
  • 1
0 votes
1 answer
29 views

Recommendation system NLP ideas

The problem: If we have a clustering problem with lets say x groups. And each group has a document describing it, lets say 3 pages. Then we have n observations each with a smaller piece of text ...
Dylan Dijk's user avatar
0 votes
0 answers
8 views

Best way to encode a tag column for clustering

I have a dataset which tells me a tech support case used a particular tech document. Every case has been tagged with which product it pertains to. Similarly tech documents are tagged with certain key ...
haldar55's user avatar
0 votes
0 answers
29 views

Predict best chess move using RNNs

I am trying to do a project with AI: in which during any certain moment of a chess game i can predict, using a RNN trained on a kaggle dataset, the best possible move that i can make. I am having ...
user3253067's user avatar
1 vote
1 answer
72 views

How does Bert masked language modelling task make sense if half the time the next sentence is wrong context in the sequence passed through the encoder

Bert has two types of tasks that it uses to learn contextual word embeddings: Masked word prediction Next sentence prediction I have read the paper and even there the training details are a little ...
figs_and_nuts's user avatar
0 votes
0 answers
44 views

How was the word2vec model trained?

Let's take the CBOW (continuous bag of words) model as the example. Suppose that, there are $c$ context words, each of which is a one-hot encoding vector. So the total number of elements of input ...
J. Doe's user avatar
  • 56
0 votes
0 answers
16 views

For a fine-tuning a transformer to type like a specific person, should I use sentence semantic embeddings or word semantic embeddings

I'm not clear on the pros and cons of each one for this particular task. Is there even a meaningful difference? My guess is using semantic embeddings for words will be better in nearly all cases ...
Austin Capobianco's user avatar
0 votes
1 answer
38 views

What does maximize average log probability mean?

In the word2vec paper (https://arxiv.org/pdf/1310.4546.pdf) that introduces the skip-gram algorithm we encounter this phrase: which says that we maximize the average log probability. Can someone help ...
Claudiu Creanga's user avatar
1 vote
0 answers
44 views

Is openAI text generation models an extension of embedding models?

we can creating embeddings using below code ...
Vinay Sharma's user avatar
1 vote
1 answer
243 views

A question about contextual embeddings in the decoder only transformer architecture (gpt)

I am reading up on the decoder only architecture Relevant excerpts: We can use any model that maps token sequences into contextual embeddings (e.g., LSTMs, Transformers): $$\phi : V^L \to R^{d \times ...
figs_and_nuts's user avatar
0 votes
1 answer
29 views

How to use location information as feature?

I have a location feature in a dataset. Some examples are: London, Uk; Sheefield Town, Ohio; UK ; North Carolina. etc. How to encode them into features? Is there any word embeddings suitable for such ...
Alex_ban's user avatar
  • 101
0 votes
0 answers
304 views

How to get Llama-2 Rotary Embeddings?

I want to get the Llama-2 rotary embeddings. I do print(model) and get the following output: In the picture I highlight the rotary embeddings. How can get the ...
Christian01's user avatar
0 votes
0 answers
32 views

"text" parameter in pinecone call from langchain

In this tutorial, I do not understand what "text" refers to vectorstore = Pinecone(index, embeddings.embed_query, "text") Could you please help?...
Karl 17302's user avatar
0 votes
1 answer
93 views

Some fundamental questions about Transformer

In the Transformer framework, a token as an input (time = $t$) $y^t$ is given by a sum of the original embedding of the token $x^t$ plus, a position embedding factor $v^t$, i.e., \begin{align} y^t = x^...
Keyflux's user avatar
  • 101
0 votes
1 answer
202 views

Appropriate input size for nn.Embedding

I’m quite new to using Pytorch and deep learning. What size of unique categories of a categorical variable is appropriate for applying the nn.Embedding ideally (best practices)? for example, if a ...
Любовь Пономарева's user avatar
0 votes
0 answers
153 views

Sum of vector sentence embeddings vs. paragraph embedding

I have been experimenting with the all-MiniLM-L6-v2 model for computing 384-dimensional vector embeddings for text paragraphs. The following code compares the embedding computed for a paragraph with ...
AlwaysLearning's user avatar
0 votes
1 answer
1k views

Training embeddings on own dataset

In my project I follow the retrieval augmented generation (RAG) approach. I want to create embeddings for my own dataset and use it in combination with llama-2. In the dataset are german annual ...
Christian01's user avatar
3 votes
1 answer
135 views

Why do semantically different words produce similar embeddings?

I am comparing words in HuggingFace web UI using e5-small-v2, one of the best vector embedding models: Assuming that the scores are in the range from 0 to 1, how ...
AlwaysLearning's user avatar
2 votes
2 answers
809 views

Why cant we use normalise position encodings instead of the cos and sine encodings used in the Transformer paper?

I'm working with Transformer models for sequence-to-sequence tasks and I'm trying to fully understand the use of positional encodings in these models. In the original "Attention is All You Need&...
mutli-arm-bandit's user avatar
1 vote
3 answers
1k views

Recommended way to embed a text thousands of tokens long?

I've split the text up sections each 512 tokens long and created embeddings for each of them. I want to combine them into 1 embedding for the full text. How do I do that? Is this even recommended? ...
codeananda's user avatar
0 votes
2 answers
67 views

building embeddings for Phrases from scratch

I have a datadet with many phrases which I would like to embed them from scratch. I dont want the cosine of the words in order to get a phrase embedding, this is because the phrases may appear in a ...
Christina Valavani's user avatar
0 votes
0 answers
45 views

Imputing Missing Text Categories in Python Using Word Embeddings/Machine Learning/LLM

I'm working on a dataset where each row represents an entity with several attributes. The dataset includes fields such as ...
nehiljain's user avatar
  • 101
1 vote
1 answer
348 views

word embeddings, what are contextual word embeddings

I m a researcher, a computational linguist and I m working on word embeddings, trying to understand them. What are contextual embeddings and how could I start working and building on my own? Thank you ...
Christina Valavani's user avatar
0 votes
1 answer
50 views

Word embeddings

I m looking into word embedding and I would like to ask if I could train words or sentences in two layers. And if I wanted that one layer is more important, how could I calculate it? For example ...
Christina Valavani's user avatar
1 vote
1 answer
41 views

ELMO embeddings

Could somebody tell me how does elmo work? Is it good for phrase embedding too? I m looking for phrase embeddings. Thank You in advance.
Christina Valavani's user avatar
0 votes
1 answer
28 views

more insights about Word2Vec implementation

As we know Word2Vec is non-contextual embedding (at word level). As per my knowledge, BOW is statistical embedding technique (word level). we can perform Word2Vec embedding in two approaches: 1. CBOW. ...
tovijayak's user avatar
0 votes
1 answer
67 views

What do averaged word vectors represent?

Assume you have high-dimensional word embeddings (d > 100) for a large number of words (|V| > 100,000) calculated over a huge non-specialized natural language corpus. Assume you have taken the ...
Hans-Peter Stricker's user avatar
0 votes
1 answer
49 views

How do we get output layer in skip-gram?

Could you please explain how do we get output layer in this architecture (vectors [0.2, 0.8, -1.4, 1.2] and ...
manabou11's user avatar
  • 101
1 vote
0 answers
54 views

I can't get good performance from BERT

I trained NLP models. This is a subset (200 instances) of my data set of 10,000 instances:This the link of the dataset on pastebin I compare an LSTM model with a glove model and a BERT model. I ...
Seydou GORO's user avatar
0 votes
0 answers
24 views

High-Cardinality Categorical Feature with frequency score

I'm working with a really high-cardinality feature as one of the inputs to my model and I'm using hash-encoded feature embedding rather than one-hot encoding. However, this method is ignoring the ...
UK-07's user avatar
  • 101
0 votes
1 answer
212 views

Text segmentation problem

I am new to ML and trying to solve problem of text segmentation. I have a transcript of news show and I want to split this transcript into parts by topic. I tried to google and asked chatgpt and found ...
Oleg Bovykin's user avatar
0 votes
0 answers
22 views

Getting synonyms for a word based on context

I want to get synonyms for a word based on it's use in sentence. for example in the sentence I will book the hotel, book is ...
Pooya Estakhri's user avatar
0 votes
1 answer
462 views

Deduplication using NLP

I have a product catalog. The user can add a new product to the catalog. The user can enter some attributes (such as color, weight, etc.) in the text boxes. The user can also mention the description ...
Shrinidhi M's user avatar
1 vote
0 answers
138 views

Otimization of similarity search for multiple embeddings by creating a weighted artificial embedding

I have embeddings of text created with a BERT model. A group of these embeddings should be used to find similar embeddings corresponding to this group. I know that you can use average or max (or ...
soph's user avatar
  • 113
3 votes
2 answers
2k views

Semantic search - combine text and image embedding

I have a question regarding combining text and image embeddings for semantic search. The use case is product search on a (B2B) marketplace, we have image(s) and title&description of the products. ...
Steven's user avatar
  • 31
0 votes
0 answers
134 views

How can I generate embeddings using previously generated BERT embeddings and feed them to an RNN?

I'm using an unlabeled news corpus to fine-tune a multi-lingual BERT model. After that I'm using those embeddings to generate embeddings for words present in a new labeled dataset. These new ...
Debbie's user avatar
  • 101
0 votes
1 answer
26 views

Why do varied delimiters on text inputs help training stability?

In the preprint paper Text and code embeddings by contrastive pre-training, the authors describe a Transformer encoder which maps the input, x and y, to embeddings, vx and vy respectively and the ...
Robbie Palmer's user avatar
0 votes
1 answer
142 views

How does softmax work for vectors?

In skipgram we predict the context words. That is the output layer before applying the softmax function is a number $V$ of words, where $V$ is the dictionary size. But each word is represented as a ...
Ruediger's user avatar
0 votes
0 answers
88 views

Better results in Document similarity using Word2Vec

I try to cluster similar support-tickets in a technical domain. The support tickets are very domain-specific and are written in various styles, lengths, using abbreviation, etc. I made a training-...
Roland's user avatar
  • 1
1 vote
1 answer
119 views

What should the numerical values of the <startofsentence> and <endofsentence> token vectors be?

I'm trying to build GPT2 from scratch. I understand how to convert each word in a sentence to its respective token index and each token is then converted to its respective word embedding vector. I ...
Austin Capobianco's user avatar
0 votes
1 answer
199 views

Doubt in ELMO, BERT, Word2Vec

I read an answer on Quora where a NLP Practioner stated that using ELMO and BERT embeddings as input to LSTM or some RNN will defeat the purpose of ELMo and BERT. I am not sure I agree with the above ...
NeverGiveUp's user avatar
2 votes
1 answer
2k views

Can I use LLM to explain codebase?

I am a Data Engineer, and I am currently assigned a task to refactor an outdated code and rectify any bugs present. However, I am unable to comprehend the code written in the existing codebase. ...
Yiffany's user avatar
  • 158

1
2 3 4 5
10