1
$\begingroup$

I've been trying to grasp the concept of Google's semantic experiences. By using it, I'm planning to implement a semantic query tool.

With universal sentence encoder I can first pre-encode all sentences and put them in the database. When user wants to perform query, input will too be converted in 512-dimensional vector, and we will perform sequential search on whole database by comparing cosine similarity (highest similarity vector is picked). But this is extremely slow...

Fortunately, on their semantic experiences page, they wrote the following:

The Universal Sentence Encoder model is very similar to what we're using in Talk to Books and Semantris, although those applications are using a dual-encoder approach that maximizes for response relevance, while the Universal Sentence Encoder is a single encoder that returns an embedding for the input, instead of a score on an input pair.


One of the simpler methods they use for transforming a sentence into embedding vector is DAN (deep averaging neural network).

From what I understand, sentence is split in words, which are converted into vectors (word2vec), then we get average of all vectors that we have obtained. Average value is finally passed to one or more hidden feedforward layers, and eventually output layer that has softmax activation functions and has 512 neurons.

How can I create a dual encoder though? Do I use two different neural networks? Or does output just contain one neuron that outputs similarity?

Thank you!

P.S

My apologies if anything ignorant has been mentioned here, I'm quite confused with the concept of projecting lower dimensional space to 512 dimensional vector space.

$\endgroup$
0
$\begingroup$

You should look into Siamese Networks, or left-right feed forward networks if you have a large set of training data such as SNLI or multiNLI datasets available to train on.

Otherwise, you should use a form of Latent semantic analysis to compress your vectors to get a faster to search set, and then perform full cosine similarity on this subset of your entire dataset.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.