2
$\begingroup$

I have two equal length vectors that come out of two identical embedding layers.

I want to calculate their similarity, and I don't trust the embedding layer enough to just use dot product (e.g. it's plausible that different coordinates are dependent wrt overall similarity). I want to learn this using examples of good and bad pairs, without actually learning the initial embedding.

What I'd like to do is to somehow combine the two vectors using another layer, and then connect this layer to an output layer to get the final decision (similar/not similar).

The trivial way is to add another layer, and fully connect the two concatenated embedding vectors to this new layer. The downside is that the model is not symmetric, which makes the search space bigger than it should be.

Is there a better way?

Ideas I have so far:

  1. Cross product the two vectors, then do the learning on the $n \times n$ output (easy enough, but might have too many weights to learn).

  2. Create two symmetrical neural nets, where the weights are symmetrical (as if mirror image of each other). Intuitively this can allow arbitrary combinations of coordinates from both sides to feed the next layer (how can equalizing the weights be done in keras?).

Is there a better way?

Bonus points: Beyond symmetry, how does one instigate transitive and reflexive properties? By just adding a ton of trivial $(x, x)$ examples?

$\endgroup$
0
$\begingroup$

That's an interesting question ! Surprisingly, I never read specific scientific papers on this particular problem. Here you want to learn a symmetric function that transforms two input vectors into a scalar. In a more general setting, we can find in the literature some needs for permutation invariant functions (see the Set Transformer and the Aggregation Schemes for Graph Neural Network ). Basically, you have two options:

  1. As you pointed out, you can take whatever non symmetric function, such as concatenation + feed forward network and train it with (x,y) and (y,x) pairs. But there might be some smarter way...
  2. Design a permutation invariant architecture. Here your chance is that you only have 2 embeddings $e_i$ and $e_j$. Sadly, you say they can't be trusted for their mutual dot product. Then, I might suggest to apply some linear/nonlinear transformation to your embeddings i.e. define a trainable matrix $Q$, and compute the dot product in the new space $v_i = Qe_i$ and $v_j = Qe_j$. By training $Q$ with example pairs $(e_i,e_j)$, you'll find a suitable transformation of the embeddings such that the dot product $v_i \cdot v_j$ fit the true similarity. You can of course use a more complex transformation (multiple non linear transformations for example). The key points is that using the same parameters for transforming both embeddings ensure the symmetry.

Bonus points: I'm not sure of your question, but my proposition in (2) might naturally deal with transitivity and reflexivity.

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.