How do people usually handle creating an embedding vector of longer texts (32000 characters?

Question

I have a set of podcast episode transcriptions in Arabic. I wish to convert these to embedding vectors so I can run a similarity comparison of them. Here's the summary statistics on the episodes:

Here's the model I used

https://huggingface.co/asafaya/bert-base-arabic

So the problem I'm running into is that the initial model I tried only accepts context windows of 512 characters. This means I can't run the whole sequence through it.

I tried chunking the text and then taking the average of the chunk vectors, but this didn't work. It seemed to create noise as all the vectors appeared similar even though their texts were not.

How do people usually handle creating an embedding vector of longer texts?

Answer 1

When dealing with longer texts, you can use a technique called "sliding window" to break the text into smaller segments. This involves taking a window of fixed size and sliding it along the text, one segment at a time. You can then concatenate the vectors of the individual segments together to form a single vector for the whole text.

Another approach is to use a hierarchical model that first encodes the text into sentence-level embeddings, and then aggregates those embeddings into a single document-level embedding.

You can also try using a transformer model that is specifically designed to handle longer sequences, such as the Longformer or the BigBird. These models are able to process sequences of up to tens of thousands of tokens, allowing you to encode entire documents in one go.