3
$\begingroup$

I found this post on Gmail's smart compose feature, and it got me thinking about trying to implement it myself.

https://ai.googleblog.com/2018/05/smart-compose-using-neural-networks-to.html

The text is super vague though - the only part that gives any direction at all is "we combined a BoW model with an RNN-LM".

I remember that BoW is an unordered representation of the input corpus where only relative magnitudes are important. But I just don't have enough experience to know how an unordered representation like this could help me predict an ordered suffix.

Does anybody have any thoughts on how this might more specifically work? Or any references for further reading? I wasn't able to find much unfortunately.

$\endgroup$
  • $\begingroup$ did you find useful content since ? $\endgroup$ – younes0 Jul 2 at 12:21
  • 1
    $\begingroup$ @younes0 I haven't. I took a break on this project but plan to start again soon. I will update the post if I learn anything :) $\endgroup$ – wheresmycookie Jul 3 at 22:28
  • $\begingroup$ This article might provide some answers. $\endgroup$ – Santino Oct 21 at 20:28
3
$\begingroup$

From the same blog,

In this hybrid approach, we encode the subject and previous email by averaging the word embeddings in each field. We then join those averaged embeddings, and feed them to the target sequence RNN-LM at every decoding step.

The BoW part of their hybrid approach is to get the general context of the email conversation by averaging the word embeddings in the subject and the previous email. Passing this vectorized representation of the general context at each step of the RNN-LM, which takes into account word order, helps get predictions that are more tailored to the subject of the conversation.

Note that at each step, the RNN-LM is getting 3 types of input -

  1. Embedding for the previous word - For immediate context.
  2. State vector - For localized context
  3. Subject and previous email averaged embeddings - For a wider, conversation-level context

Hopefully, it is clearer now, that the model incorporates order with the use of RNN-LM and the BoW part is to condition the output to the subject of the conversation.

$\endgroup$
  • 1
    $\begingroup$ I was also trying to understand that paper but didn't understood how the data is prepared before feeding into their architecture. Can you shed some light on it. What is their X and Y? $\endgroup$ – user_12 Nov 20 at 11:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.