I have a set of podcast episode transcriptions in Arabic. I wish to convert these to embedding vectors so I can run a similarity comparison of them. Here's the summary statistics on the episodes:
Here's the model I used
So the problem I'm running into is that the initial model I tried only accepts context windows of 512 characters. This means I can't run the whole sequence through it.
I tried chunking the text and then taking the average of the chunk vectors, but this didn't work. It seemed to create noise as all the vectors appeared similar even though their texts were not.
How do people usually handle creating an embedding vector of longer texts?