Questions tagged [word-embeddings]
Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.
501
questions
0
votes
0
answers
16
views
Approach for Multi-class Classification of texts
I'm trying to do a project where I have paragraphs and I need to classify them into multiple labels. The dataset is around 40k rows with labels.
I understand there is no one right approach but should ...
1
vote
1
answer
23
views
How can I use contextual embeddings with BERT for sentiment analysis/classification
I have a BERT model which I want to use for sentiment analysis/classification. E.g. I have some tweets that need to get a POSITIVE,NEGATIVE or NEUTRAL label. I can't understand how contextual ...
0
votes
0
answers
12
views
Accuracy decreased after using google word2vec model for a sentiment classification [NLP][word-embedding]
I am using Amazon fine food reviews for a sentiment classification project. while I used my dataset corpus to train avg word2vec , I was getting an accuracy of 89 %.
by using BOW and TF-IDF, i was ...
0
votes
0
answers
24
views
What is the state of art production search algorithm right now for semantic search? LSH? or other clustering method?
Trying to implement semantic search for high cardinality embedding for my own learning purpose, so far LSH seems promising, but I am wondering what is the state of algorithm big tech company are using ...
0
votes
1
answer
29
views
Recommendation system NLP ideas
The problem:
If we have a clustering problem with lets say x groups. And each group has a document describing it, lets say 3 pages. Then we have n observations each with a smaller piece of text ...
0
votes
0
answers
8
views
Best way to encode a tag column for clustering
I have a dataset which tells me a tech support case used a particular tech document.
Every case has been tagged with which product it pertains to.
Similarly tech documents are tagged with certain key ...
0
votes
0
answers
24
views
Predict best chess move using RNNs
I am trying to do a project with AI: in which during any certain moment of a chess game i can predict, using a RNN trained on a kaggle dataset, the best possible move that i can make.
I am having ...
1
vote
1
answer
69
views
How does Bert masked language modelling task make sense if half the time the next sentence is wrong context in the sequence passed through the encoder
Bert has two types of tasks that it uses to learn contextual word embeddings:
Masked word prediction
Next sentence prediction
I have read the paper and even there the training details are a little ...
0
votes
0
answers
42
views
How was the word2vec model trained?
Let's take the CBOW (continuous bag of words) model as the example.
Suppose that, there are $c$ context words, each of which is a one-hot encoding vector. So the total number of elements of input ...
0
votes
0
answers
16
views
For a fine-tuning a transformer to type like a specific person, should I use sentence semantic embeddings or word semantic embeddings
I'm not clear on the pros and cons of each one for this particular task. Is there even a meaningful difference?
My guess is using semantic embeddings for words will be better in nearly all cases ...
0
votes
1
answer
35
views
What does maximize average log probability mean?
In the word2vec paper (https://arxiv.org/pdf/1310.4546.pdf) that introduces the skip-gram algorithm we encounter this phrase:
which says that we maximize the average log probability.
Can someone help ...
0
votes
0
answers
39
views
Is openAI text generation models an extension of embedding models?
we can creating embeddings using below code
...
1
vote
1
answer
177
views
A question about contextual embeddings in the decoder only transformer architecture (gpt)
I am reading up on the decoder only architecture
Relevant excerpts:
We can use any model that maps token sequences into contextual embeddings (e.g., LSTMs, Transformers):
$$\phi : V^L \to R^{d \times ...
0
votes
1
answer
28
views
How to use location information as feature?
I have a location feature in a dataset. Some examples are: London, Uk; Sheefield Town, Ohio; UK ; North Carolina. etc. How to encode them into features? Is there any word embeddings suitable for such ...
0
votes
0
answers
294
views
How to get Llama-2 Rotary Embeddings?
I want to get the Llama-2 rotary embeddings. I do print(model) and get the following output:
In the picture I highlight the rotary embeddings.
How can get the ...
0
votes
0
answers
30
views
"text" parameter in pinecone call from langchain
In this tutorial, I do not understand what "text" refers to
vectorstore = Pinecone(index, embeddings.embed_query, "text")
Could you please help?...
0
votes
1
answer
86
views
Some fundamental questions about Transformer
In the Transformer framework, a token as an input (time = $t$) $y^t$ is given by a sum of the original embedding of the token $x^t$ plus, a position embedding factor $v^t$, i.e.,
\begin{align}
y^t = x^...
0
votes
1
answer
109
views
Appropriate input size for nn.Embedding
I’m quite new to using Pytorch and deep learning. What size of unique categories of a categorical variable is appropriate for applying the nn.Embedding ideally (best practices)? for example, if a ...
0
votes
0
answers
103
views
Sum of vector sentence embeddings vs. paragraph embedding
I have been experimenting with the all-MiniLM-L6-v2 model for computing 384-dimensional vector embeddings for text paragraphs. The following code compares the embedding computed for a paragraph with ...
0
votes
1
answer
815
views
Training embeddings on own dataset
In my project I follow the retrieval augmented generation (RAG) approach.
I want to create embeddings for my own dataset and use it in combination with llama-2.
In the dataset are german annual ...
3
votes
1
answer
117
views
Why do semantically different words produce similar embeddings?
I am comparing words in HuggingFace web UI using e5-small-v2, one of the best vector embedding models:
Assuming that the scores are in the range from 0 to 1, how ...
2
votes
2
answers
593
views
Why cant we use normalise position encodings instead of the cos and sine encodings used in the Transformer paper?
I'm working with Transformer models for sequence-to-sequence tasks and I'm trying to fully understand the use of positional encodings in these models.
In the original "Attention is All You Need&...
1
vote
2
answers
1k
views
Recommended way to embed a text thousands of tokens long?
I've split the text up sections each 512 tokens long and created embeddings for each of them.
I want to combine them into 1 embedding for the full text. How do I do that? Is this even recommended?
...
0
votes
2
answers
59
views
building embeddings for Phrases from scratch
I have a datadet with many phrases which I would like to embed them from scratch. I dont want the cosine of the words in order to get a phrase embedding, this is because the phrases may appear in a ...
0
votes
0
answers
41
views
Imputing Missing Text Categories in Python Using Word Embeddings/Machine Learning/LLM
I'm working on a dataset where each row represents an entity with several attributes. The dataset includes fields such as ...
1
vote
1
answer
140
views
word embeddings, what are contextual word embeddings
I m a researcher, a computational linguist and I m working on word embeddings, trying to understand them.
What are contextual embeddings and how could I start working and building on my own?
Thank you ...
0
votes
1
answer
48
views
Word embeddings
I m looking into word embedding and I would like to ask if I could train words or sentences in two layers. And if I wanted that one layer is more important, how could I calculate it? For example ...
1
vote
1
answer
36
views
ELMO embeddings
Could somebody tell me how does elmo work?
Is it good for phrase embedding too?
I m looking for phrase embeddings.
Thank You in advance.
0
votes
1
answer
26
views
more insights about Word2Vec implementation
As we know Word2Vec is non-contextual embedding (at word level). As per my knowledge, BOW is statistical embedding technique (word level).
we can perform Word2Vec embedding in two approaches: 1. CBOW. ...
0
votes
1
answer
62
views
What do averaged word vectors represent?
Assume you have high-dimensional word embeddings (d > 100) for a large number of words (|V| > 100,000) calculated over a huge non-specialized natural language corpus. Assume you have taken the ...
0
votes
1
answer
46
views
How do we get output layer in skip-gram?
Could you please explain how do we get output layer in this architecture (vectors [0.2, 0.8, -1.4, 1.2] and ...
1
vote
0
answers
54
views
I can't get good performance from BERT
I trained NLP models. This is a subset (200 instances) of my data set of 10,000 instances:This the link of the dataset on pastebin
I compare an LSTM model with a glove model and a BERT model. I ...
0
votes
0
answers
19
views
High-Cardinality Categorical Feature with frequency score
I'm working with a really high-cardinality feature as one of the inputs to my model and I'm using hash-encoded feature embedding rather than one-hot encoding. However, this method is ignoring the ...
0
votes
1
answer
167
views
Text segmentation problem
I am new to ML and trying to solve problem of text segmentation.
I have a transcript of news show and I want to split this transcript into parts by topic. I tried to google and asked chatgpt and found ...
0
votes
0
answers
22
views
Getting synonyms for a word based on context
I want to get synonyms for a word based on it's use in sentence. for example in the sentence I will book the hotel, book is ...
0
votes
1
answer
404
views
Deduplication using NLP
I have a product catalog.
The user can add a new product to the catalog. The user can enter some attributes (such as color, weight, etc.) in the text boxes. The user can also mention the description ...
1
vote
0
answers
130
views
Otimization of similarity search for multiple embeddings by creating a weighted artificial embedding
I have embeddings of text created with a BERT model. A group of these embeddings should be used to find similar embeddings corresponding to this group. I know that you can use average or max (or ...
3
votes
2
answers
1k
views
Semantic search - combine text and image embedding
I have a question regarding combining text and image embeddings for semantic search. The use case is product search on a (B2B) marketplace, we have image(s) and title&description of the products. ...
0
votes
0
answers
129
views
How can I generate embeddings using previously generated BERT embeddings and feed them to an RNN?
I'm using an unlabeled news corpus to fine-tune a multi-lingual BERT model. After that I'm using those embeddings to generate embeddings for words present in a new labeled dataset. These new ...
0
votes
1
answer
25
views
Why do varied delimiters on text inputs help training stability?
In the preprint paper Text and code embeddings by contrastive pre-training, the authors describe a Transformer encoder which
maps the input, x and y, to embeddings, vx and vy respectively and the ...
0
votes
0
answers
37
views
Max word embedding
I'm trying to understand the concept behind Max Word / Sentence embeddings also in contrast to mean word embedding. They used it for example in this article. So far I didn't find good resources which ...
0
votes
1
answer
119
views
How does softmax work for vectors?
In skipgram we predict the context words. That is the output layer before applying the softmax function is a number $V$ of words, where $V$ is the dictionary size. But each word is represented as a ...
0
votes
0
answers
43
views
Generate vector database for userdata
I need a point in the right direction for the problem I'm trying to solve:
I have a lot of already classified short articles. The articles themselves or a reference to them should be stored in some ...
0
votes
0
answers
112
views
Clustering with BERT. Why are my clusters overlapped? How to improve BERT embeddings?
I am trying to create BERT embeddings of text data, then use dimensionality reduction and cluster. I tried with some big datasets like amazon reviews and 20newsgroups, but whenever I created ...
0
votes
0
answers
22
views
Align sub-sentences with sentences using embeddings
I would like ask for ideas
So I have sub-sentences, embedded in the context of their respective full-sentences. Then, I have other full-sentences and I would like to find
a) if they have similar sub-...
0
votes
0
answers
18
views
Creating Word Embeddings using BERT for Machine-Generated Text Data
I have a dataset of machine-generated sequences that are not natural language, but the order of the words in the sequence is important. I want to create word embeddings using BERT to capture the ...
0
votes
0
answers
88
views
Better results in Document similarity using Word2Vec
I try to cluster similar support-tickets in a technical domain. The support tickets are very domain-specific and are written in various styles, lengths, using abbreviation, etc.
I made a training-...
1
vote
1
answer
97
views
What should the numerical values of the <startofsentence> and <endofsentence> token vectors be?
I'm trying to build GPT2 from scratch. I understand how to convert each word in a sentence to its respective token index and each token is then converted to its respective word embedding vector. I ...
0
votes
1
answer
183
views
Doubt in ELMO, BERT, Word2Vec
I read an answer on Quora where a NLP Practioner stated that using ELMO and BERT embeddings as input to LSTM or some RNN will defeat the purpose of ELMo and BERT. I am not sure I agree with the above ...
0
votes
0
answers
13
views
Can different corpora sizes in word embedding models over time effect the results? If so, can bootstrapped sampling avoid this?
I have a word embedding model with glove algorithm where I compare association of word X with Y over time, using fourr discreet periods. However, size of my data varies for each period. In Period one ...