Here's the code from my notebook:

%tensorflow_version 1.x
import tensorflow as tf
import tensorflow_hub as hub

elmo = hub.Module("https://tfhub.dev/google/elmo/2", trainable=True)

def elmo_vectors(x):
    embeddings = elmo(x, signature="default", as_dict=True)["elmo"]
    with tf.Session() as sess:
        return sess.run(embeddings)

Output for non-English language: (Hindi in this example)

words = ['गोकुल']
v = elmo_vectors(words)
print(v.shape) # (1,1,1024)
# Output: [ 0.3731584   0.5700774  -0.48072845 ... -0.1241736   0.5961436 -0.6986947 ]

The documentation of the pre-trained ELMo on Tensorflow Hub shows that it was trained only on the English language.
That is, the dataset from 1 billion word benchmark is based on monolingual English data. (Source)

So, how/why am I getting embeddings for non-English vocabulary words from ELMo using the TF Hub model?


While ELMo was trained on English data, it does not know whether the data you give it as input is English or not.

The input of ELMo is received at character-level. It may happen that the 1B Word data had hindi characters intermixed, case in which your characters would be encoded as they are or, most probably, your characters are encoded as unknown characters (just like the unknown token <unk> for word-level NLP but for characters).

ELMo is just a bunch of mathematical operations, so it takes whatever it receives and computes its operations on it, first, taking the character embedding with the characters you pass to it, then with a char-CNN followed by two highway layers and finally a bidirectional LSTM.

  • $\begingroup$ The input of ELMo is received at character-level. It seems that the tokenizer for ELMo is Moses Tokenizer (example), which is not character-level tokenization. $\endgroup$ – Gokul NC Jun 25 '20 at 13:26
  • $\begingroup$ The input to ELMo is tokenized first for words and then, for each word, encoded as characters. The input tensor is of dimensions [batch_size, max_seq_length, max_char_length], where max_char_length is by default 50, that is words are assumed to be at most 50 characters long. You can verify this in the original ELMo article or in the very TF Hub documentation, where is says "Computes contextualized word representations using character-based word representations and..." $\endgroup$ – noe Jun 25 '20 at 16:19
  • $\begingroup$ The second dimension of the batch indexes the word, while the third dimension indexes the character within the word. $\endgroup$ – noe Jun 25 '20 at 16:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.