Questions tagged [word-embeddings]
Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.
501 questions
0
votes
2
answers
70
views
0
votes
1
answer
41
views
Word embedding is not getting better
I created a simple neural network to train the word embeddings.
I have 6 tokens only: ["apple", "banana", "lime", "red", "yellow", "green"].
...
- 33
0
votes
0
answers
19
views
How to broadly structure a Record Linkage model for Emails on top of a Vector Embedding model for Semantic Search with entities?
Sorry the broad and naive question, but the structure I have in mind is as follows:
Extract the text from a large collection of Documents with varying types.
This part I plan to use Apache Tika and ...
- 101
0
votes
0
answers
21
views
NLP techniques to count job skills match
In our job portal search we are performing search by using dense embedding of Job Title + Skills across a pool of candidates. After this first step of retreival I'm planning to perform rerank based on ...
- 111
0
votes
0
answers
120
views
Recommendation System: Two-Tower Model Underperforming Simple Embedding Average Baseline
I'm trying to build a recommendation on a dataset of product purchases. The dataset consists of roughly 4 Amazon products that a particular user has bought (in sequence). I want to use the first 3 ...
- 1
2
votes
1
answer
67
views
Intuition behind g variable calculation in the original word2vec implementation
I am trying to develop the intuition of word2vec training.
Looking into the word2vec source code, I see (for example, in skip-gram):
...
- 121
0
votes
1
answer
25
views
Clustering method for frequency embeddings
I have, for example, the following lists of words that I want to cluster. The lists could have different lengths, and the vocabulary is $W = \{a,b,c\}$. The criteria of clustering 2 lists into a same ...
0
votes
0
answers
40
views
google news word2vec - 3 million 300-dimension vectors, but only 6908 distinct dimension numbers - why?
This google news word2vec dataset:
https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit?resourcekey=0-wjGZdNAUop6WykTtMip30g
google news word2vec - supposedly has has 3 million 300-...
- 101
0
votes
0
answers
7
views
Recreating Text Embeddings From An Example Dataset
I am in a situation where I have a list of sentences, and a list of their ideal embeddings on a 25-dimensional vector. I am trying to use a neural network to generate new encodings, but I am ...
- 13
0
votes
0
answers
19
views
Do a transformer's embeddings self-organise the same way as word2vec embeddings?
Word2vec embeddings are well-known for being able to do vector arithmetic on them. So King - Queen ≈ Man - Woman. Or Germany - Berlin ≈ France - Paris.
When I first learned about transformers, one of ...
- 1,254
0
votes
0
answers
30
views
Should multiple categorical embeddings be combined for a conditional GAN (cGAN)?
I'm trying to make a conditional GAN (cGAN) that generates YouTube thumbnails based on a title and a video category/genre.
It's not working whatsoever, not even close, and so I'm trying to go back to ...
- 21
0
votes
0
answers
23
views
How contextual embeddings learned during training a transformer are applied to the input sequence at inference time
I'm trying to understand contextual word embeddings better, and how they are applied at inference time.
While training a transformer, embeddings are learned as parameters during training. Are the ...
2
votes
1
answer
648
views
How OpenAI embeddings work?
I was looking at the Stanford CS224N NLP with Deep Learning lecture, and in the first two videos, we are introduced to word2vec models. The high-level idea mentioned was that we have a 'big corpus' of ...
1
vote
0
answers
802
views
"No sentence-transformers model found with name" on huggingface even though it exists
I am trying to use infgrad/stella-base-en-v2 on hugging to generate embeddings using langchain
The model exists on the huggingface hub
The model is listed on the MTEB leaderboard
The model has ...
- 903
1
vote
0
answers
26
views
word2vec predicts the same word for all inputs
i build word2vec network with 2 linear layers from pytorch. for every word as an input i consistently train model to predict words before and after, for example: i was visiting my grandma's house, for ...
- 73
0
votes
0
answers
22
views
Insights about W0rd2Vec
As per my knowledge, Word2Vec is belongs to non-contextual embedding technique. this have only semantic relationship between words.
We can implement Word2Vec, either in CBoW or skip-gram model. but i ...
- 43
1
vote
1
answer
790
views
How to combine two vector embeddings into one?
I want to use OpenCLIP for generating embeddings for each slide in an array of pptx presentations.
To improve the quality of the results, I want to vectorize both slide text content and preview images....
- 113
0
votes
0
answers
31
views
Approach for Multi-class Classification of texts
I'm trying to do a project where I have paragraphs and I need to classify them into multiple labels. The dataset is around 40k rows with labels.
I understand there is no one right approach but should ...
1
vote
1
answer
296
views
How can I use contextual embeddings with BERT for sentiment analysis/classification
I have a BERT model which I want to use for sentiment analysis/classification. E.g. I have some tweets that need to get a POSITIVE,NEGATIVE or NEUTRAL label. I can't understand how contextual ...
0
votes
0
answers
24
views
Accuracy decreased after using google word2vec model for a sentiment classification [NLP][word-embedding]
I am using Amazon fine food reviews for a sentiment classification project. while I used my dataset corpus to train avg word2vec , I was getting an accuracy of 89 %.
by using BOW and TF-IDF, i was ...
0
votes
0
answers
26
views
What is the state of art production search algorithm right now for semantic search? LSH? or other clustering method?
Trying to implement semantic search for high cardinality embedding for my own learning purpose, so far LSH seems promising, but I am wondering what is the state of algorithm big tech company are using ...
- 1
0
votes
1
answer
35
views
Recommendation system NLP ideas
The problem:
If we have a clustering problem with lets say x groups. And each group has a document describing it, lets say 3 pages. Then we have n observations each with a smaller piece of text ...
- 101
0
votes
0
answers
13
views
Best way to encode a tag column for clustering
I have a dataset which tells me a tech support case used a particular tech document.
Every case has been tagged with which product it pertains to.
Similarly tech documents are tagged with certain key ...
- 1
0
votes
0
answers
38
views
Predict best chess move using RNNs
I am trying to do a project with AI: in which during any certain moment of a chess game i can predict, using a RNN trained on a kaggle dataset, the best possible move that i can make.
I am having ...
2
votes
1
answer
169
views
How does Bert masked language modelling task make sense if half the time the next sentence is wrong context in the sequence passed through the encoder
Bert has two types of tasks that it uses to learn contextual word embeddings:
Masked word prediction
Next sentence prediction
I have read the paper and even there the training details are a little ...
- 903
0
votes
1
answer
51
views
What does maximize average log probability mean?
In the word2vec paper (https://arxiv.org/pdf/1310.4546.pdf) that introduces the skip-gram algorithm we encounter this phrase:
which says that we maximize the average log probability.
Can someone help ...
- 198
1
vote
0
answers
54
views
Is openAI text generation models an extension of embedding models?
we can creating embeddings using below code
...
- 187
1
vote
1
answer
373
views
A question about contextual embeddings in the decoder only transformer architecture (gpt)
I am reading up on the decoder only architecture
Relevant excerpts:
We can use any model that maps token sequences into contextual embeddings (e.g., LSTMs, Transformers):
$$\phi : V^L \to R^{d \times ...
- 903
0
votes
1
answer
32
views
How to use location information as feature?
I have a location feature in a dataset. Some examples are: London, Uk; Sheefield Town, Ohio; UK ; North Carolina. etc. How to encode them into features? Is there any word embeddings suitable for such ...
- 101
1
vote
0
answers
36
views
"text" parameter in pinecone call from langchain
In this tutorial, I do not understand what "text" refers to
vectorstore = Pinecone(index, embeddings.embed_query, "text")
Could you please help?...
- 49
1
vote
1
answer
156
views
Some fundamental questions about Transformer
In the Transformer framework, a token as an input (time = $t$) $y^t$ is given by a sum of the original embedding of the token $x^t$ plus, a position embedding factor $v^t$, i.e.,
\begin{align}
y^t = x^...
- 111
0
votes
1
answer
445
views
Appropriate input size for nn.Embedding
I’m quite new to using Pytorch and deep learning. What size of unique categories of a categorical variable is appropriate for applying the nn.Embedding ideally (best practices)? for example, if a ...
0
votes
1
answer
2k
views
Training embeddings on own dataset
In my project I follow the retrieval augmented generation (RAG) approach.
I want to create embeddings for my own dataset and use it in combination with llama-2.
In the dataset are german annual ...
- 121
3
votes
1
answer
213
views
Why do semantically different words produce similar embeddings?
I am comparing words in HuggingFace web UI using e5-small-v2, one of the best vector embedding models:
Assuming that the scores are in the range from 0 to 1, how ...
- 141
2
votes
2
answers
1k
views
Why cant we use normalise position encodings instead of the cos and sine encodings used in the Transformer paper?
I'm working with Transformer models for sequence-to-sequence tasks and I'm trying to fully understand the use of positional encodings in these models.
In the original "Attention is All You Need&...
1
vote
3
answers
2k
views
Recommended way to embed a text thousands of tokens long?
I've split the text up sections each 512 tokens long and created embeddings for each of them.
I want to combine them into 1 embedding for the full text. How do I do that? Is this even recommended?
...
- 288
0
votes
2
answers
90
views
building embeddings for Phrases from scratch
I have a datadet with many phrases which I would like to embed them from scratch. I dont want the cosine of the words in order to get a phrase embedding, this is because the phrases may appear in a ...
1
vote
1
answer
698
views
word embeddings, what are contextual word embeddings
I m a researcher, a computational linguist and I m working on word embeddings, trying to understand them.
What are contextual embeddings and how could I start working and building on my own?
Thank you ...
0
votes
1
answer
72
views
Word embeddings
I m looking into word embedding and I would like to ask if I could train words or sentences in two layers. And if I wanted that one layer is more important, how could I calculate it? For example ...
1
vote
1
answer
55
views
ELMO embeddings
Could somebody tell me how does elmo work?
Is it good for phrase embedding too?
I m looking for phrase embeddings.
Thank You in advance.
0
votes
1
answer
30
views
more insights about Word2Vec implementation
As we know Word2Vec is non-contextual embedding (at word level). As per my knowledge, BOW is statistical embedding technique (word level).
we can perform Word2Vec embedding in two approaches: 1. CBOW. ...
- 67
0
votes
1
answer
148
views
What do averaged word vectors represent?
Assume you have high-dimensional word embeddings (d > 100) for a large number of words (|V| > 100,000) calculated over a huge non-specialized natural language corpus. Assume you have taken the ...
0
votes
1
answer
72
views
How do we get output layer in skip-gram?
Could you please explain how do we get output layer in this architecture (vectors [0.2, 0.8, -1.4, 1.2] and ...
- 101
1
vote
0
answers
61
views
I can't get good performance from BERT
I trained NLP models. This is a subset (200 instances) of my data set of 10,000 instances:This the link of the dataset on pastebin
I compare an LSTM model with a glove model and a BERT model. I ...
- 131
0
votes
1
answer
328
views
Text segmentation problem
I am new to ML and trying to solve problem of text segmentation.
I have a transcript of news show and I want to split this transcript into parts by topic. I tried to google and asked chatgpt and found ...
0
votes
1
answer
592
views
Deduplication using NLP
I have a product catalog.
The user can add a new product to the catalog. The user can enter some attributes (such as color, weight, etc.) in the text boxes. The user can also mention the description ...
- 401
1
vote
0
answers
162
views
Otimization of similarity search for multiple embeddings by creating a weighted artificial embedding
I have embeddings of text created with a BERT model. A group of these embeddings should be used to find similar embeddings corresponding to this group. I know that you can use average or max (or ...
- 113
3
votes
2
answers
3k
views
Semantic search - combine text and image embedding
I have a question regarding combining text and image embeddings for semantic search. The use case is product search on a (B2B) marketplace, we have image(s) and title&description of the products. ...
- 31
0
votes
0
answers
197
views
How can I generate embeddings using previously generated BERT embeddings and feed them to an RNN?
I'm using an unlabeled news corpus to fine-tune a multi-lingual BERT model. After that I'm using those embeddings to generate embeddings for words present in a new labeled dataset. These new ...
- 101
0
votes
1
answer
29
views
Why do varied delimiters on text inputs help training stability?
In the preprint paper Text and code embeddings by contrastive pre-training, the authors describe a Transformer encoder which
maps the input, x and y, to embeddings, vx and vy respectively and the ...
