0
$\begingroup$

Here's the code from my notebook:

%tensorflow_version 1.x
import tensorflow as tf
import tensorflow_hub as hub

elmo = hub.Module("https://tfhub.dev/google/elmo/2", trainable=True)
tf.logging.set_verbosity(tf.logging.ERROR)

def elmo_vectors(x):
    embeddings = elmo(x, signature="default", as_dict=True)["elmo"]
    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())
        sess.run(tf.tables_initializer())
        return sess.run(embeddings)

Output for non-English language: (Hindi in this example)

words = ['गोकुल']
v = elmo_vectors(words)
print(v.shape) # (1,1,1024)
print(v[0][0])
# Output: [ 0.3731584   0.5700774  -0.48072845 ... -0.1241736   0.5961436 -0.6986947 ]

The documentation of the pre-trained ELMo on Tensorflow Hub shows that it was trained only on the English language.
That is, the dataset from 1 billion word benchmark is based on monolingual English data. (Source)

So, how/why am I getting embeddings for non-English vocabulary words from ELMo using the TF Hub model?

$\endgroup$
2
  • $\begingroup$ Please consider upvoting and marking it as correct the answer that you find it useful. $\endgroup$
    – noe
    Commented Nov 23, 2020 at 15:22
  • $\begingroup$ I had actually asked that question here too: How does pre-trained ELMo produce embeddings for characters of different languages? (out-of-vocabulary alphabets). Basically, they use byte sequence encoding, meaning that, it could have been tokenized as ASCII characters (bytes). Hence the non-English Unicode characters are probably taken as multiple ASCII characters or not. (Please correct me if I'm wrong.) $\endgroup$
    – Gokul NC
    Commented Nov 24, 2020 at 10:18

1 Answer 1

1
$\begingroup$

While ELMo was trained on English data, it does not know whether the data you give it as input is English or not.

The input of ELMo is received at character-level. It may happen that the 1B Word data had hindi characters intermixed, case in which your characters would be encoded as they are or, most probably, your characters are encoded as unknown characters (just like the unknown token <unk> for word-level NLP but for characters).

ELMo is just a bunch of mathematical operations, so it takes whatever it receives and computes its operations on it, first, taking the character embedding with the characters you pass to it, then with a char-CNN followed by two highway layers and finally a bidirectional LSTM.

$\endgroup$
3
  • $\begingroup$ The input of ELMo is received at character-level. It seems that the tokenizer for ELMo is Moses Tokenizer (example), which is not character-level tokenization. $\endgroup$
    – Gokul NC
    Commented Jun 25, 2020 at 13:26
  • $\begingroup$ The input to ELMo is tokenized first for words and then, for each word, encoded as characters. The input tensor is of dimensions [batch_size, max_seq_length, max_char_length], where max_char_length is by default 50, that is words are assumed to be at most 50 characters long. You can verify this in the original ELMo article or in the very TF Hub documentation, where is says "Computes contextualized word representations using character-based word representations and..." $\endgroup$
    – noe
    Commented Jun 25, 2020 at 16:19
  • $\begingroup$ The second dimension of the batch indexes the word, while the third dimension indexes the character within the word. $\endgroup$
    – noe
    Commented Jun 25, 2020 at 16:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.