Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I've split the text up sections each 512 tokens long and created embeddings for each of them.
I want to combine them into 1 embedding for the full text. How do I do that? Is this even recommended?
AdaV2 has max token limit of 8k, I thought about concatenating each embedding like str(embed1) + str(embed2) + ... and stopping when it reaches 8k tokens. But each embed is 15k tokens if converted to a string.
One way is to just take the embedding of the CLS token. Another way is to average all the tokens.
Key search terms are sentence embeddings and document embeddings.
The best models are specifically trained (or fine-tuned) to give good sentence embeddings. I believe InferSent and SBERT are considered current state-of-the-art?
There are also multilingual sentence-embedding models, e.g. https://arxiv.org/abs/2007.01852 (and see https://arxiv.org/abs/2302.08387 for a follow-up making them smaller).
You can easily combine embeddings by taking the element-wise mean of them. You can do more fancy things like building attention models but this is a good way to start.
import numpy as np
a = [1, 2, 3]
b = [10, 20, 30]
np.mean([a, b], axis=0)
# [ 5.5, 11., 16.5]