Sequence to sequence RNN model, maximum number of training size

Question

So when running this example script from Keras repo (https://github.com/keras-team/keras/blob/master/examples/lstm_seq2seq.py), I found that we can easily run into out of memory for the input or output one-hot encoding in this code:

encoder_input_data = np.zeros(
    (len(input_texts), max_encoder_seq_length, num_encoder_tokens),
    dtype='float32')
decoder_input_data = np.zeros(
    (len(input_texts), max_decoder_seq_length, num_decoder_tokens),
    dtype='float32')
decoder_target_data = np.zeros(
    (len(input_texts), max_decoder_seq_length, num_decoder_tokens),
dtype='float32')

When I have a huge size of training samples, this one-hot does not fit into memory. Is there way to handle this issue?

UPDATE: I changed float32 to uint, but thats about the smallest one hot array can get.

Answer 1

so I found a solution by dividing input training data into small batches and it did the trick. Here is the code:

divideby=1000
for j in range(divideby):
    start=j*len(input_texts)/divideby
    end=(j+1)*len(input_texts)/divideby if j<divideby+1 else -1

    encoder_input_data = np.zeros(
        (end-start, max_encoder_seq_length, num_encoder_tokens),
        dtype='uint8')  # size_of_training_samples, max_length_word_measured_in_characters,number_of_unique_chars,
    decoder_input_data = np.zeros(
        (end-start, max_decoder_seq_length, num_decoder_tokens),
        dtype='uint8')
    decoder_target_data = np.zeros(
        (end-start, max_decoder_seq_length, num_decoder_tokens),
        dtype='uint8')

    for i, (input_text, target_text) in enumerate(zip(input_texts[start:end], target_texts[start:end])):
        for t, char in enumerate(input_text.split(splitby)):
            encoder_input_data[i, t, input_token_index[char]] = 1.
        for t, char in enumerate(target_text.split(splitby)):
            # decoder_target_data is ahead of decoder_input_data by one timestep
            decoder_input_data[i, t, target_token_index[char]] = 1.
            if t > 0:
                # decoder_target_data will be ahead by one timestep
                # and will not include the start character.
                decoder_target_data[i, t - 1, target_token_index[char]] = 1.

    model.fit([encoder_input_data, decoder_input_data], decoder_target_data,
              batch_size=batch_size,
              epochs=epochs,
              validation_split=0.2)
```