It seems that Embedding vector is the best solution here, however you need to also consider hash collision if the dict is growing.

It seems that Embedding vector is the best solution here.

However, you may consider a variant of the one-hot encoding called 'one-hot hashing trick". In this variant, when the number of unique words is too large to be assigned a unique index in a dictionary, one may hash words of into vector of fixed size.

One advantage in your use case is that you may perform online encoding. If you have not encountered every vocabulary words yet, you may still assign a hash. Then later, new words may be added to the vocabulary. One pitfall though is "hash collisions". Indeed there is a probability that two different words end up with the same hash.

I found an example of this hashing trick in the excellent "Deep learning with Python" from François Chollet.

import numpy as np
samples = ['The cat sat on the mat.', 'The dog ate my homework.']
dimensionality = 1000
max_length = 10

results = np.zeros((len(samples), max_length, dimensionality))
for i, sample in enumerate(samples):
    for j, word in list(enumerate(sample.split()))[:max_length]:
        # the word is given a hash with a random integer
        index = abs(hash(word)) % dimensionality
        results[i, j, index] = 1.0

The resulting array:

results.shape
(2, 10, 1000)

You can observe that common words between the two sentences are given the same index: The in position 0 of the two sentences (positions 0 and 6 of dimension 2).

np.where(results > 0)
(array([0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1], dtype=int64),
 array([0, 1, 2, 3, 4, 5, 0, 1, 2, 3, 4], dtype=int64),
 array([ 23,  58, 467, 442,  83,  77,  23, 798, 618, 301, 942], dtype=int64))