Trying to implement a "smart compose" feature

Question

I found this post on Gmail's smart compose feature, and it got me thinking about trying to implement it myself.

https://ai.googleblog.com/2018/05/smart-compose-using-neural-networks-to.html

The text is super vague though - the only part that gives any direction at all is "we combined a BoW model with an RNN-LM".

I remember that BoW is an unordered representation of the input corpus where only relative magnitudes are important. But I just don't have enough experience to know how an unordered representation like this could help me predict an ordered suffix.

Does anybody have any thoughts on how this might more specifically work? Or any references for further reading? I wasn't able to find much unfortunately.

Answer 1

From the same blog,

In this hybrid approach, we encode the subject and previous email by averaging the word embeddings in each field. We then join those averaged embeddings, and feed them to the target sequence RNN-LM at every decoding step.

The BoW part of their hybrid approach is to get the general context of the email conversation by averaging the word embeddings in the subject and the previous email. Passing this vectorized representation of the general context at each step of the RNN-LM, which takes into account word order, helps get predictions that are more tailored to the subject of the conversation.

Note that at each step, the RNN-LM is getting 3 types of input -

1. Embedding for the previous word - For immediate context.
2. State vector - For localized context
3. Subject and previous email averaged embeddings - For a wider, conversation-level context

Hopefully, it is clearer now, that the model incorporates order with the use of RNN-LM and the BoW part is to condition the output to the subject of the conversation.

Answer 2

For next Phrase prediction, using RNNs to train a model on your own data would be more beneficial than using pre-trained models. And vice versa for Next Word prediction. This might help: [here][https://towardsdatascience.com/gmail-style-smart-compose-using-char-n-gram-language-models-a73c09550447]

For understanding what will be X and Y for training, read about teacher forcing technique: [Teacher enforcing technique][https://machinelearningmastery.com/teacher-forcing-for-recurrent-neural-networks/]

I would also suggest you to use multiple models to get desired results, which would include Next Word Prediction Model, Next phrase prediction model and possibly a Word/Phrase Completion model. A combination of these models can achieve great results.

There are various models with high accuracy(low perplexity) for next word prediction which can be used in there native form. Also refer [Tranformers from Hugging Face][https://github.com/huggingface/transformers], it might be of great help.