# How to build a symmetric similarity model on top of embeddings?

I have two equal length vectors that come out of two identical embedding layers.

I want to calculate their similarity, and I don't trust the embedding layer enough to just use dot product (e.g. it's plausible that different coordinates are dependent wrt overall similarity). I want to learn this using examples of good and bad pairs, without actually learning the initial embedding.

What I'd like to do is to somehow combine the two vectors using another layer, and then connect this layer to an output layer to get the final decision (similar/not similar).

The trivial way is to add another layer, and fully connect the two concatenated embedding vectors to this new layer. The downside is that the model is not symmetric, which makes the search space bigger than it should be.

Is there a better way?

Ideas I have so far:

1. Cross product the two vectors, then do the learning on the $$n \times n$$ output (easy enough, but might have too many weights to learn).

2. Create two symmetrical neural nets, where the weights are symmetrical (as if mirror image of each other). Intuitively this can allow arbitrary combinations of coordinates from both sides to feed the next layer (how can equalizing the weights be done in keras?).

Is there a better way?

Bonus points: Beyond symmetry, how does one instigate transitive and reflexive properties? By just adding a ton of trivial $$(x, x)$$ examples?

2. Design a permutation invariant architecture. Here your chance is that you only have 2 embeddings $$e_i$$ and $$e_j$$. Sadly, you say they can't be trusted for their mutual dot product. Then, I might suggest to apply some linear/nonlinear transformation to your embeddings i.e. define a trainable matrix $$Q$$, and compute the dot product in the new space $$v_i = Qe_i$$ and $$v_j = Qe_j$$. By training $$Q$$ with example pairs $$(e_i,e_j)$$, you'll find a suitable transformation of the embeddings such that the dot product $$v_i \cdot v_j$$ fit the true similarity. You can of course use a more complex transformation (multiple non linear transformations for example). The key points is that using the same parameters for transforming both embeddings ensure the symmetry.