3
$\begingroup$

I have a question regarding combining text and image embeddings for semantic search. The use case is product search on a (B2B) marketplace, we have image(s) and title&description of the products. I want to allow the user to search both the image and the text but I’m not sure how to combine them. The idea is that the combination of the third party product description and images holds more information than the parts - which we know is true for a lot products, especially those with low description quality.

My current idea is to use a CLIP model to embed the image and CLIP/Sentence Transformer model to embed the text, apply the same to the query and concatenate the two vectors. If both of the embeddings are scaled to unit length then they should have the same weight and impact the final similarity the same. But I see that this approach can be quite limited as I’m taking two embeddings and just smashing them together without regard for any nuance.

  • An extra question - if I were to use CLIP for both the description and image, could I average (pool) the embeddings? They are from the same vector space so it could work.

The main question, do you have experience with this approach? Is there something I should be aware of? I found several sources online that validate my idea - here, here and here - but none are directly related to semantic search. I’m interested in some feedback/experience with this approach.

And an interesting follow-up - are there good alternatives to this approach? Based on my research I have a few ideas but I would appreciate feedback/experience with those as well

  • Search for products via image and text similarity separately and then look at highest combined similarity (we could just sum the two similarity scores)
  • Train a custom model that combines the two embeddings but I’m not quite sure how to go about that since I don’t have a target. Can I approach this like a Cross-Encoder and just put the concatenated product embedding and concatenated query embedding in a model?
  • PCA or any dimensionality reduction

The more complex approaches are out of scope for me for now. We’re not ready for an ML model in production search engineering-wise. But I would appreciate any tips/resources to learn about this option as well since we might get there soon-ish.

Any tips/resources/experiences are very much appreciated! N.B.: I haven’t found much about this use case online besides this - https://discuss.huggingface.co/t/similarity-search-with-combined-image-and-text/19168

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

Currently I am working in a marketplace too, and I am trying to combine text and embedding features. I am doing this by simply concatenation of them. It's important to normalize the text features and embeddings separately to achieve better results with cosine similarity.

If you want you can add weight between your embeddings, but if you want unit distance (in range [0;1]) you should normalise again after reweighting.

I think concatenation is way more convenient, than separate search. You can store vectors via one storage, there may also be problems with the union of two results, for example, if these sets are disjoint.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks! I was originally planning of using this approach as well but part of the reason I didn't do it were limitations of the vector DB we use - Weaviate. It's great to hear this a valid approach, I couldn't think of a reason this shouldn't work but wasn't sure I wasn't missing something $\endgroup$
    – Steven
    Commented Jun 23, 2023 at 14:07
0
$\begingroup$

I consulted this question with a few people and the recommendation was to go with

  • Search for products via image and text similarity separately and then look at combined similarity (we could just sum/average the two similarity scores)

as it should be the easiest and most flexible option. My initial idea should most likely work as well but it is less flexible.

Hopefully this helps somebody in the future.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.