I have a question regarding combining text and image embeddings for semantic search. The use case is product search on a (B2B) marketplace, we have image(s) and title&description of the products. I want to allow the user to search both the image and the text but I’m not sure how to combine them. The idea is that the combination of the third party product description and images holds more information than the parts - which we know is true for a lot products, especially those with low description quality.
My current idea is to use a CLIP model to embed the image and CLIP/Sentence Transformer model to embed the text, apply the same to the query and concatenate the two vectors. If both of the embeddings are scaled to unit length then they should have the same weight and impact the final similarity the same. But I see that this approach can be quite limited as I’m taking two embeddings and just smashing them together without regard for any nuance.
- An extra question - if I were to use CLIP for both the description and image, could I average (pool) the embeddings? They are from the same vector space so it could work.
The main question, do you have experience with this approach? Is there something I should be aware of? I found several sources online that validate my idea - here, here and here - but none are directly related to semantic search. I’m interested in some feedback/experience with this approach.
And an interesting follow-up - are there good alternatives to this approach? Based on my research I have a few ideas but I would appreciate feedback/experience with those as well
- Search for products via image and text similarity separately and then look at highest combined similarity (we could just sum the two similarity scores)
- Train a custom model that combines the two embeddings but I’m not quite sure how to go about that since I don’t have a target. Can I approach this like a Cross-Encoder and just put the concatenated product embedding and concatenated query embedding in a model?
- PCA or any dimensionality reduction
The more complex approaches are out of scope for me for now. We’re not ready for an ML model in production search engineering-wise. But I would appreciate any tips/resources to learn about this option as well since we might get there soon-ish.
Any tips/resources/experiences are very much appreciated! N.B.: I haven’t found much about this use case online besides this - https://discuss.huggingface.co/t/similarity-search-with-combined-image-and-text/19168