Questions tagged [embeddings]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
17 views

Do categorical embeddings leak data in time series?

I am a bit confused on this matter, I can't find any resources that touch on the following but my logic says that embeddings do introduce data leakage in time series: Considering a temporal dataset ...
idontknowmuch's user avatar
0 votes
0 answers
18 views

Content based filtering with dissimilar user interests

I'm prototyping a system for recommending articles to users. A single embedding vector is generated for the article summary, and users self report a list of non-enumerated interests. Concretely, a ...
stevester94's user avatar
1 vote
1 answer
55 views

How is openAI embedding models trained?

how it the embedding model trained? Are the embeddings simply extracted from chatGPT4 or are they trained differently from the beginning (pre-training stage)?
haneulkim's user avatar
  • 469
0 votes
1 answer
40 views

How to combine two vector embeddings into one?

I want to use OpenCLIP for generating embeddings for each slide in an array of pptx presentations. To improve the quality of the results, I want to vectorize both slide text content and preview images....
Olek Gornostal's user avatar
1 vote
1 answer
66 views

Why Transformer applies Dropout after Positional Encoding?

Why Transformers applies Dropout after Positional Encoding? Attention Is All You Need Not sure what is the benefit of removing 10% of tokens in a sequence by default. Read Why use dropout in ...
mon's user avatar
  • 711
0 votes
0 answers
19 views

Rather than build a classification model, Is building an embedding regression model feasible?

Face recognition models like VGG Face are designed to have a classification head on top and then trained to classify face images, but after they are trained the classification head can be removed and ...
Ahmed Gamal's user avatar
0 votes
0 answers
20 views

Dimenson reduction from a cosine similarity matrix

I have a silly little question: I have 200 press articles (string), I vectorize these articles with an embedding model (sentence embedding), so I have 1024 values per article. I then have a 200 x 1024 ...
Bertrand's user avatar
1 vote
1 answer
30 views

How can I use contextual embeddings with BERT for sentiment analysis/classification

I have a BERT model which I want to use for sentiment analysis/classification. E.g. I have some tweets that need to get a POSITIVE,NEGATIVE or NEUTRAL label. I can't understand how contextual ...
average_discrete_math_enjoyer's user avatar
0 votes
1 answer
32 views

How can we make embeddings from the text?

I have a dataset with questions and answers to them. I want to make embeddings of questions and save them in a vector database. Next, I will make a query to the database. With the help of the pinecone ...
7wafer7's user avatar
1 vote
1 answer
33 views

How to add a new item in the embeddings vocabulary?

Imagine you have trained a model containing an Embedding layer. Your model performs well and you're happy with your embedding. Then, suddenly, you want to add a new item in your vocabulary. In other ...
MarcoM's user avatar
  • 111
0 votes
0 answers
15 views

How are GPT2 token embedding vectors processed internally?

I am experimenting with the GPT2-XL model and trying to understand the internal structure. While I understand most of the components and how they affect the size of the activation tensors (such as ...
John Doe's user avatar
2 votes
0 answers
48 views

Why does cross-attention in an NMT decoder use the encoder embeddings as values?

In the Vaswani 2017 paper introducing encoder-decoder transformers, the cross-attention step in the decoder is visualised as follows: Because keys and values are always taken to be equal, this figure ...
Mew's user avatar
  • 233
1 vote
0 answers
16 views

Similarity search with text and tabular data

If I have two documents, D1 and D2 and a function f which computes the (normalized) document ...
CutePoison's user avatar
0 votes
0 answers
64 views

Calculating correlation between embedding features

I am looking for a technique to calculate correlation for embedding features (array of floats). I'm interested in the correlation between features (embedding-embedding) as well as between feature and ...
Drew Serles's user avatar
0 votes
0 answers
16 views

For a fine-tuning a transformer to type like a specific person, should I use sentence semantic embeddings or word semantic embeddings

I'm not clear on the pros and cons of each one for this particular task. Is there even a meaningful difference? My guess is using semantic embeddings for words will be better in nearly all cases ...
Austin Capobianco's user avatar
0 votes
0 answers
11 views

Handling Irregular Time Series Data with Multiple Categories in Hourly Resolution Embeddings

I'm working with a time series dataset that has 100 categories for a single feature, and I've generated embeddings for these categories. However, my problem arises when I am trying to sample the data ...
Mohammad Asif Ibna Mustafa's user avatar
1 vote
0 answers
44 views

Is openAI text generation models an extension of embedding models?

we can creating embeddings using below code ...
Vinay Sharma's user avatar
0 votes
0 answers
28 views

Using Embedding For Regularization

Is using embeddings for regularization a valid practice? My reasoning for that is that encoding training/tests datasets into smaller vectors would allow a smaller network with fewer parameters and ...
Adenilson Arcanjo's user avatar
1 vote
1 answer
101 views

Why is T5 often used in text-to-data for text prompt encoders?

In the text-to-data(music, image, audio, etc.) generative AI field, one method of encoding text prompts is to use pre-trained language models. Such an approach was used in research on Moûsai [1] and ...
NakataKoo's user avatar
0 votes
0 answers
52 views

Combining Textual, Categorical and Numerical data for Semantic Search using SentenceTransformers model

I'm building a food semantic search model and I want to use a pre-trained SentenceTransformers model with cosine similarity. I'm using Epicurious dataset for the corpus which consists of textual (&...
Alex's user avatar
  • 1
0 votes
1 answer
28 views

Using text embeddings directly to compute similarity vs using them as features for a model that predicts similariy

Say you have a problem where you have a query and a set of result documents and you want to rank the result documents according to the query. Say also you have embeddings for the query and for the ...
user1893354's user avatar
1 vote
0 answers
118 views

CLIP Visual Transformer image encoder

I was doing some experiments with the CLIP's visual transformer encoder output (clip-ViT-B-32). So basically given the same scene or image, it should output almost ...
Mary's user avatar
  • 217
0 votes
1 answer
123 views

Are the terms "vector embedding" and "embedding vector" interchangeable?

Googling "vector embedding" gives me ~40k results, including from Pinecone, Weaviate, and OpenAI. Googling ...
Alex Hall's user avatar
  • 103
0 votes
0 answers
22 views

Are Vector embeddings position or direction vectors?

I've seen statements about embeddings being positions on a hypersphere. I've also seen statements about using dot product to determine similarity of these positions. This implies to me that we care ...
iasksillyquestions's user avatar
0 votes
0 answers
903 views

How to add knowledge to the LLM using LangChain (at a high level)?

At a super high level, I would like to create a fantasy language AI tutor. For this question, however, I would like to better understand how, generally speaking, you add your own custom data/knowledge ...
Lance's user avatar
  • 273
2 votes
2 answers
718 views

Why cant we use normalise position encodings instead of the cos and sine encodings used in the Transformer paper?

I'm working with Transformer models for sequence-to-sequence tasks and I'm trying to fully understand the use of positional encodings in these models. In the original "Attention is All You Need&...
mutli-arm-bandit's user avatar
1 vote
3 answers
1k views

Recommended way to embed a text thousands of tokens long?

I've split the text up sections each 512 tokens long and created embeddings for each of them. I want to combine them into 1 embedding for the full text. How do I do that? Is this even recommended? ...
codeananda's user avatar
0 votes
0 answers
107 views

Why are similarities all close to 1 in my two tower recommender?

I am trying to build a simple two tower recommender system on the MovieLens 100k dataset. The user tower is just a simple embedding layer. The item tower uses an embedding layer and concats that with ...
ds_'s user avatar
  • 1
0 votes
1 answer
31 views

How do I interpret low dimentional embeddings of high dimentional embeddings?

I am trying to understand what I am supposed to learn about a problem when using dimensionality reduction methods. In particular, I am referring to methods like t-SNE and UMAP. For the most part I am ...
Finncent Price's user avatar
0 votes
0 answers
22 views

High-Cardinality Categorical Feature with frequency score

I'm working with a really high-cardinality feature as one of the inputs to my model and I'm using hash-encoded feature embedding rather than one-hot encoding. However, this method is ignoring the ...
UK-07's user avatar
  • 101
1 vote
0 answers
133 views

Otimization of similarity search for multiple embeddings by creating a weighted artificial embedding

I have embeddings of text created with a BERT model. A group of these embeddings should be used to find similar embeddings corresponding to this group. I know that you can use average or max (or ...
soph's user avatar
  • 113
0 votes
0 answers
39 views

Max word embedding

I'm trying to understand the concept behind Max Word / Sentence embeddings also in contrast to mean word embedding. They used it for example in this article. So far I didn't find good resources which ...
soph's user avatar
  • 113
0 votes
0 answers
58 views

Distance Metric for a dataset with embeddings and numerical columns

I'm trying to build an Approximate Nearest Neighbours model that can fetch similar records in a dataset, that are contextually similar. For example, for a record of job=Software Engineer and age=25, a ...
Augustine Samuel's user avatar
0 votes
0 answers
23 views

Align sub-sentences with sentences using embeddings

I would like ask for ideas So I have sub-sentences, embedded in the context of their respective full-sentences. Then, I have other full-sentences and I would like to find a) if they have similar sub-...
aqua's user avatar
  • 123
0 votes
1 answer
264 views

SBERT Embeddings from Conversations

I have a dataset consisting of text-based conversations between two humans. One conversation has on average 20 turns and can look as follows: ...
DictionaryProver's user avatar
1 vote
1 answer
47 views

Visualizing convolutional neural networks embedding

In this article, the author creates a graph (at the end of the post) from the embeddings of different words found by transformer model. I would like to do a similar thing for a convolutional neural ...
Zan's user avatar
  • 113
0 votes
1 answer
36 views

Is there a way to embed curves into smaller vectors?

I know about embeddings for words, but I would like to know if it is possible to do something similar for curves. What I mean by curves is a curve of a function. Say I have 1000 points corresponding ...
WLD's user avatar
  • 1
1 vote
1 answer
73 views

Which pre-trained model to select to generate embeddings from shop names written in English?

Good afternoon! I have a dataset with thousands of shop names written in English. Several shop names might belong to one business entity, for instance, shops with names "KFC 001", "WWW....
rsx's user avatar
  • 11
0 votes
1 answer
23 views

Are there methods to represent entire knowledge graphs using a single embedding vector?

In a knowledge graph, embedding vectors can be learned for nodes (node embedding) and edges (edge embeddings). Is there a method to learn one single embedding vector for the entire knowledge graph?
skumaravel's user avatar
1 vote
0 answers
78 views

Machine learning with mixed variables dataset (numerical, categorical and embeddings)

I'm working on a machine learning project where I'm trying to predict the revenue of a movie. My dataset contains mixed data types. There are numerical features (rating, number of votes, release year,....
Mathieu Rousseau's user avatar
1 vote
0 answers
164 views

Embedding performing unexpectedly worse than multi-hot encoding

I'm pretty new to ML and have set up a Keras model with a number of categorical features of users & items to predict positive reactions. Each feature is a list of IDs from a database - i.e. ...
Tom's user avatar
  • 11
0 votes
0 answers
58 views

Practical application of denoising autoencoders

I have been reading into autoencoders for the purpose of denoising data. In the examples i found (eg. [1, 2, 3], which are the first few google results) they have the following input/output: Input ...
Leander Moesinger's user avatar
1 vote
1 answer
64 views

Understand the reason of embedding and the size inside it in Pytorch

I'm very new to pytorch - taking a course in udemy. There is something I find hard to understand and would like to get explaination about, in a bit simpler words than what I can find in the ...
Reut's user avatar
  • 349
1 vote
2 answers
97 views

Biometrics identification with embeddings comparison and "unknown"/"other" class/label

This is a general or more conceptual questions about biometric classification models, based on deep learning neural networks. The goal of the system is to take a set of features (e.g. voice recording, ...
Triceratops's user avatar
1 vote
1 answer
93 views

Product embeddings

Hei, I have a list of purchase baskets from customers and would like to build embeddings for the products. For example: BASKET1 = ['PRODUCT234', 'PRODUCT214', 'PRODUCT768'] BASKET2 = ['PRODUCT2', '...
ryuzakinho's user avatar
1 vote
1 answer
164 views

Is there a sensible notion of 'character embeddings'?

There are several popular word embeddings available (e.g., Fasttext and GloVe); In short, those embeddings are a tool to encode words along with a sensible notion of semantics attached to those words (...
Ramiro Hum-Sah's user avatar
1 vote
0 answers
299 views

How are the embedding and context matrices created and updated in word embedding?

I am struggling to understand how word embedding works, especially how the embedding matrix $W$ and context matrix $W'$ are created/updated. I understand that in the Input we may have a one-hot ...
Revolucion for Monica's user avatar
2 votes
1 answer
2k views

Transformer time series classification using time2vec positional embedding

I want to use a transformer model to do classification of fixed-length time series. I was following along this tutorial using keras which uses time2vec as a positional embedding. According to the ...
Reignbeaux's user avatar
1 vote
1 answer
744 views

Do we perform text embedding before or after train-test splitting?

Do we perform text embedding before or after train-test splitting? I know that for encoding variables, usually done after the split. However, I'm not sure if that's also the case for text processing?
Student's user avatar
  • 411
2 votes
0 answers
20 views

Dicussion for X-vectors

I am posting this question to ask some questions regarding the X-vector embedding proposed by Synder et al. The paper is X-VECTORS: ROBUST DNN EMBEDDINGS FOR SPEAKER RECOGNITION With reference to the ...
Leo's user avatar
  • 125