Similarity with respect to a specific concept in text embeddings

Question

In text embeddings, cosine similarity is often used to find texts similar to a search query. However, I don't want to find a text that is overall similar, but similar with regards to a specific concept (which I can also embed).

Example: Let's say you have many movie reviews that you have embedded. You choose one review and want to search for similar ones with regard to the cinematography only.

More formally stated, my problem is the following: Let $x_1, \dots, x_n \in \mathbb{R}^d$ vector embeddings of texts $t_1, \dots, t_n$. Further, let $q \in \mathbb{R}^d$ be the embedding of a search query. I want to rank the texts by similarity with the search query with regards to a specific concept/aspect with embedding $k \in \mathbb{R}^d$.

I thought about using the embedding $k$ of the concept, like "cinematography" and then projecting the text embeddings onto that direction $$S_C \propto \langle P_k q, P_k x_i\rangle$$

but that does not make sense as cosine similarity looks at the angle.

One could project just one of the text embeddings $$S_C \propto \langle P_k q, x_i\rangle = S_c(k, q) \cdot S_c(k, x_i)$$ but this amounts to multiplying the cosine similarities of the concept with both text embeddings.

I did not find any research literature on this specific problem. Is there a way to align the cosine similarity to a specific concept/context?

Answer 1

You can generate custom embeddings for your corpus/dataset and then calculate the cosine similarity. When you generate your own word embeddings for your dataset, words with similar meaning will be close to each other in vector space. So even if two words don't have any letter in common but are in same context would have some value of cosine similarity. Below is the link of a tensorflow tutorial of generating custom embeddings - https://www.tensorflow.org/text/guide/word_embeddings

Answer 2

Could you treat this like a classification problem? As in, assume that the concepts have subclasses and consider texts to be similar if they are in the same class?

It's hard to pin down specific concepts in an embedding space. The dimensions are effectively meaningless, but you can get some information by comparing embeddings with other embeddings in the space. The Euclidean properties of an embedding space do sometimes hold information about specific concepts; like the classic example in word embeddings where ('King' - 'Man' + 'Woman') ≈ 'Queen' suggests that the vector ('King' - 'Queen') is something like a masculine/feminine axis. But this is oversimplified, because the words King and Queen aren't used in identical contexts, so the difference in their meaning isn't just a single concept.