How to combine two different embeddings in the best way possible?

Question

I have two models which are giving two books embedding

Ml_model_a => book1_embedding [ 1, 200 ]

Ml_model_b => book2_embedding [ 1, 200 ]

I am building a third model which will take these two different embeddings to tell me which book to choose.

Now my final layer is the classification between 0,1 ( which book to choose ). How to learn these embeddings best way possible to classify better?

What I have tried yet :

If I am averaging those embeddings and then sending to one model, then embeddings are losing a lot of information, so I am using concatenation method.

But it's not classifying well, Is there any other model, a technique I could use to enhance the capability of learning to book embedding and predict which book to take?

Answer 1

I would say that the best thing to do here is to concatenate both embeddings and use the concatenated vector as an input for your binary classification model without using the norm -> you loose way too much information, i.e.:

final_layer = concatenate [ book_1_attention, book_2_attention]

I reckon that concatenating without norm will increase the dimension (but that's potentially a problem for which you have solutions - such as regularisation)

Answer 2

You can have a look into the literature about learning to rank, for example this work, which ensures reflexivity, is antisymmetric and transitive.

Answer 3

If you apply the l2 norm to the vectors you loose a lot of informations, i think the best way to act is to evaluate these options:

• concatenate embeddings with/out a linear layer on top of that
• sum the embeddings with/out a linear layer on top of that