1
$\begingroup$

I've split the text up sections each 512 tokens long and created embeddings for each of them.

I want to combine them into 1 embedding for the full text. How do I do that? Is this even recommended?

AdaV2 has max token limit of 8k, I thought about concatenating each embedding like str(embed1) + str(embed2) + ... and stopping when it reaches 8k tokens. But each embed is 15k tokens if converted to a string.

$\endgroup$

3 Answers 3

1
$\begingroup$

One way is to just take the embedding of the CLS token. Another way is to average all the tokens.

Key search terms are sentence embeddings and document embeddings.

The best models are specifically trained (or fine-tuned) to give good sentence embeddings. I believe InferSent and SBERT are considered current state-of-the-art?

There are also multilingual sentence-embedding models, e.g. https://arxiv.org/abs/2007.01852 (and see https://arxiv.org/abs/2302.08387 for a follow-up making them smaller).

$\endgroup$
0
$\begingroup$

You can easily combine embeddings by taking the element-wise mean of them. You can do more fancy things like building attention models but this is a good way to start.

import numpy as np

a = [1, 2, 3]
b = [10, 20, 30]

np.mean([a, b], axis=0)

# [ 5.5, 11.,  16.5]
$\endgroup$
0
$\begingroup$

You may use different ways to combine them together, but averaging would be the simplest where you may loose some details and enhance common concepts in the list that you are combining. I would recommend to check out text embeddings with larger tokens such as Instructor

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.