Here's the code from my notebook:

%tensorflow_version 1.x
import tensorflow as tf
import tensorflow_hub as hub

elmo = hub.Module("https://tfhub.dev/google/elmo/2", trainable=True)

def elmo_vectors(x):
    embeddings = elmo(x, signature="default", as_dict=True)["elmo"]
    with tf.Session() as sess:
        return sess.run(embeddings)

Output for non-English language: (Hindi in this example)

words = ['गोकुल']
v = elmo_vectors(words)
print(v.shape) # (1,1,1024)
# Output: [ 0.3731584   0.5700774  -0.48072845 ... -0.1241736   0.5961436 -0.6986947 ]

The documentation of the pre-trained ELMo on Tensorflow Hub shows that it was trained only on the English language.
That is, the dataset from 1 billion word benchmark is based on monolingual English data. (Source)

So, how/why am I getting embeddings for non-English vocabulary words from ELMo using the TF Hub model?

  • $\begingroup$ This question is off-topic here because this is a programming issue and general programming issues are off-topic here. Please, read ai.stackexchange.com/help/on-topic to know more about what kind of question you can ask on this site. I will migrate this post to Data Science SE. $\endgroup$ – nbro Jun 25 '20 at 11:30
  • $\begingroup$ Please consider upvoting and marking it as correct the answer that you find it useful. $\endgroup$ – ncasas Nov 23 '20 at 15:22
  • $\begingroup$ I had actually asked that question here too: How does pre-trained ELMo produce embeddings for characters of different languages? (out-of-vocabulary alphabets). Basically, they use byte sequence encoding, meaning that, it could have been tokenized as ASCII characters (bytes). Hence the non-English Unicode characters are probably taken as multiple ASCII characters or not. (Please correct me if I'm wrong.) $\endgroup$ – Gokul NC Nov 24 '20 at 10:18

While ELMo was trained on English data, it does not know whether the data you give it as input is English or not.

The input of ELMo is received at character-level. It may happen that the 1B Word data had hindi characters intermixed, case in which your characters would be encoded as they are or, most probably, your characters are encoded as unknown characters (just like the unknown token <unk> for word-level NLP but for characters).

ELMo is just a bunch of mathematical operations, so it takes whatever it receives and computes its operations on it, first, taking the character embedding with the characters you pass to it, then with a char-CNN followed by two highway layers and finally a bidirectional LSTM.

  • $\begingroup$ The input of ELMo is received at character-level. It seems that the tokenizer for ELMo is Moses Tokenizer (example), which is not character-level tokenization. $\endgroup$ – Gokul NC Jun 25 '20 at 13:26
  • $\begingroup$ The input to ELMo is tokenized first for words and then, for each word, encoded as characters. The input tensor is of dimensions [batch_size, max_seq_length, max_char_length], where max_char_length is by default 50, that is words are assumed to be at most 50 characters long. You can verify this in the original ELMo article or in the very TF Hub documentation, where is says "Computes contextualized word representations using character-based word representations and..." $\endgroup$ – ncasas Jun 25 '20 at 16:19
  • $\begingroup$ The second dimension of the batch indexes the word, while the third dimension indexes the character within the word. $\endgroup$ – ncasas Jun 25 '20 at 16:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.