I'm currently trying to find a way of loading/deserializing a .json file containing Flair word embeddings that is too large to fit in my RAM at once (>60GB .json with 32GB of RAM). My current code for loading the embedding is below.

def get_embedding_table(config):
    words_id2vec = json.load(open(config.words_id2vector_filename, 'r'))
    words_vectors = [0] * len(words_id2vec)
    for id, vec in words_id2vec.items():
        words_vectors[int(id)] = vec

    words_vectors.append(list(np.random.uniform(0, 1, config.embedding_dim)))
    words_embedding_table = tf.Variable(name='words_emb_table', initial_value=words_vectors, dtype=tf.float32)

The rest of the code that I am trying to reproduce with a different word embedding can be found here.

I wonder if it is somehow possible to access the embedding table without deserialization of the entire .json file, for example: by sequentially reading it, somehow splitting it, or reading it directly from my disk. I would greatly appreciate your input!

  • $\begingroup$ Maybe this can help? stackoverflow.com/q/10715628/891919 $\endgroup$ – Erwan Oct 22 '19 at 23:14
  • $\begingroup$ @Erwan I've looked into this but I am not sure how I could make TensorFlow work with this approach. Do you maybe have an idea about this? $\endgroup$ – Nels Oct 23 '19 at 17:21
  • $\begingroup$ I don't know I'm afraid. I'm lucky enough to have access to a high performance cluster, so I don't have this kind of problem! If you're in academia you could maybe explore if there are any such resources you could use? $\endgroup$ – Erwan Oct 23 '19 at 22:46

There are a couple of options:

  1. Incrementally parse JSON with something like ijson. Munge and append each part to tf.Variable.

  2. Reduce the precision of the numbers dtype=tf.float32 to dtype=tf.bfloat16

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.