Autocomplete with deep learning

Question

I got interested in autocompletion using deep learning and tutorials that I found where conditioned always on specific number of characters (given 40 characters predict the next character or the whole complete word). But in real world the autocomplete is done after entering a first letter already.

I was checking out how autocompletion works in Whatsapp I had following observations:
#1. Autocomplete starts working already after a first letter.
#2. Before I would enter a space, Whatsapp suggests the continuation of a word, example: entering 'L' would result in suggesting 'L', 'Like', 'Last'.
#3. If I enter a space after a word, autocomplete would suggest me whole next word, example: entering 'Last ' would result in suggesting 'time', 'night', 'week'.
#4. If I enter 'I am ru' it will suggest 'running', 'run'. (Pay attention: grammatically, run is wrong)

One could use RNNs/LSTMs with one-hot encoding or word-embedding to do such things.

However, I wondered what the train dataset should look like:

1. is it char-based model with different n-grams words, example:
   1.1 char-based 1-gram ('l', 'i', 'k', 'e') word would cover the cases #1 and #2 because I would be able to sample the next character.
   1.2. char-based with 2-gram word ('l', 'i', 'k', 'e', ' ', 's','u','n') would cover #3 as I would be able to predict the whole complete next word (character-based) and it also preserves the dependence on the previous word.

By using 1.1 and 1.2 I could cover cases #1, #2 and #3 and model 2-words dependence.

2. is it word-based model with different n-grams, example:
   2.1 word-based with 1 grams, eg. "like sun" -> sequence: "like ", predict: "sun". This would allow me to model #3
   2.2 word-based with 2 grams, eg. "like sun and" -> sequence: "like sun ", predict: "and". This would allow me to model a bit longer dependence.

By using 1.1 and 2.1 I would cover cases #1, #2, #3.

Questions:
I wonder what would be better to do: use 1.1 and 1.2 or 1.1 and 2.1.

Also, let me know if my way of thinking is correct or if I am completely wrong somewhere or if I miss something or if you think there are some easier way of doing it. Would be glad about any suggestions.

Answer 1

This can be easily worked out using a standard Seq2Seq word-level model without any modifications in the training process. During inference, in the decoder, just mask the vocabulary based on the partial word input which has been provided.

By masking, I mean to assign a very high score (inf) to words which don't start with the partial word input, thus constraining the decoder to predict words which only start with the partial word input.