How to dual encode two sentences to show similarity score

Question

I've been trying to grasp the concept of Google's semantic experiences. By using it, I'm planning to implement a semantic query tool.

With universal sentence encoder I can first pre-encode all sentences and put them in the database. When user wants to perform query, input will too be converted in 512-dimensional vector, and we will perform sequential search on whole database by comparing cosine similarity (highest similarity vector is picked). But this is extremely slow...

Fortunately, on their semantic experiences page, they wrote the following:

The Universal Sentence Encoder model is very similar to what we're using in Talk to Books and Semantris, although those applications are using a dual-encoder approach that maximizes for response relevance, while the Universal Sentence Encoder is a single encoder that returns an embedding for the input, instead of a score on an input pair.

---

One of the simpler methods they use for transforming a sentence into embedding vector is DAN (deep averaging neural network).

From what I understand, sentence is split in words, which are converted into vectors (word2vec), then we get average of all vectors that we have obtained. Average value is finally passed to one or more hidden feedforward layers, and eventually output layer that has softmax activation functions and has 512 neurons.

How can I create a dual encoder though? Do I use two different neural networks? Or does output just contain one neuron that outputs similarity?

Thank you!

P.S

My apologies if anything ignorant has been mentioned here, I'm quite confused with the concept of projecting lower dimensional space to 512 dimensional vector space.

Answer 1

How can I create a dual encoder though? Do I use two different neural networks? Or does output just contain one neuron that outputs similarity?

You don't need to create two different neural networks (or two different encoders) for it. Whenever you have a symmetric pieces problem, like in your case, its two sentence similarity, you should encode both the sentences using the same encoder and then use cosine similarity. (It can be the similarity of two faces also - just the way FaceNet is expected to be used.)

we will perform sequential search on whole database by comparing cosine similarity (highest similarity vector is picked). But this is extremely slow...

You are right, I am assuming you would be using a for/while loop for computing the similarity. In computing the vector similarity of one sentence against a database of thousands or millions, you should make use of a high-performance library like BLAS which would do the same calculations sub-second.

Facebook three years ago came out with a faster matrix library called FAISS. This would reduce your speed drastically.

Answer 2

You should look into Siamese Networks, or left-right feed forward networks if you have a large set of training data such as SNLI or multiNLI datasets available to train on.

Otherwise, you should use a form of Latent semantic analysis to compress your vectors to get a faster to search set, and then perform full cosine similarity on this subset of your entire dataset.