0
$\begingroup$

I want to use OpenCLIP for generating embeddings for each slide in an array of pptx presentations. To improve the quality of the results, I want to vectorize both slide text content and preview images. So then I can run queries like "What slides show the project plan of Fintech project delivery in the form of a Gantt chart?", assuming text embeddings will cover the "Fintech" part and image embeddings the "Gantt chart" part.

Is that even possible?

If so, can I just concatenate two vectors?

$\endgroup$

1 Answer 1

2
$\begingroup$

How do you plan on using these embeddings?

You can definitely use concatenated embeddings for similarity/retrieval, but only when comparing concat embeddings to other concat embeddings.

Your point about running text queries makes it sound like you want to run text based retrieval of slides. This wouldn't work with concat embeddings, because your query text vector won't be the same size as your concat embeddings.

Concat embeddings only work if you plan on doing retrieval queries with other concat embeddings.

If you want to be able to retrieve slides with text queries (or slide text with image queries), you need to have text and image embeddings projected into the same space.

In theory OpenCLIP models should do this out of the box, but I have a hunch that powerpoint images are different enough from the standard CLIP model training data that you would get bad performance.

To get around this, you want to train a small model (MLP or even linear layer) to map OpenCLIP embeddings to new embeddings. Train this with the standard CLIP loss, just on your own specific dataset. This will give you embeddings that work much better for your data (powerpoints) compared to out of the box CLIP models.

$\endgroup$
1
  • $\begingroup$ "Concat embeddings only work if you plan on doing retrieval queries with other concat embeddings." and "I have a hunch that powerpoint images are different enough from the standard CLIP model training data" -- This makes so much sense. I could've realized this myself :) Thank you for answering! $\endgroup$ Commented Mar 23 at 16:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.