0
$\begingroup$

So when running this example script from Keras repo (https://github.com/keras-team/keras/blob/master/examples/lstm_seq2seq.py), I found that we can easily run into out of memory for the input or output one-hot encoding in this code:

encoder_input_data = np.zeros(
    (len(input_texts), max_encoder_seq_length, num_encoder_tokens),
    dtype='float32')
decoder_input_data = np.zeros(
    (len(input_texts), max_decoder_seq_length, num_decoder_tokens),
    dtype='float32')
decoder_target_data = np.zeros(
    (len(input_texts), max_decoder_seq_length, num_decoder_tokens),
dtype='float32')

When I have a huge size of training samples, this one-hot does not fit into memory. Is there way to handle this issue?

UPDATE: I changed float32 to uint, but thats about the smallest one hot array can get.

$\endgroup$
0
$\begingroup$

so I found a solution by dividing input training data into small batches and it did the trick. Here is the code:

divideby=1000
for j in range(divideby):
    start=j*len(input_texts)/divideby
    end=(j+1)*len(input_texts)/divideby if j<divideby+1 else -1

    encoder_input_data = np.zeros(
        (end-start, max_encoder_seq_length, num_encoder_tokens),
        dtype='uint8')  # size_of_training_samples, max_length_word_measured_in_characters,number_of_unique_chars,
    decoder_input_data = np.zeros(
        (end-start, max_decoder_seq_length, num_decoder_tokens),
        dtype='uint8')
    decoder_target_data = np.zeros(
        (end-start, max_decoder_seq_length, num_decoder_tokens),
        dtype='uint8')

    for i, (input_text, target_text) in enumerate(zip(input_texts[start:end], target_texts[start:end])):
        for t, char in enumerate(input_text.split(splitby)):
            encoder_input_data[i, t, input_token_index[char]] = 1.
        for t, char in enumerate(target_text.split(splitby)):
            # decoder_target_data is ahead of decoder_input_data by one timestep
            decoder_input_data[i, t, target_token_index[char]] = 1.
            if t > 0:
                # decoder_target_data will be ahead by one timestep
                # and will not include the start character.
                decoder_target_data[i, t - 1, target_token_index[char]] = 1.

    model.fit([encoder_input_data, decoder_input_data], decoder_target_data,
              batch_size=batch_size,
              epochs=epochs,
              validation_split=0.2)
```
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.