LSTM classification with different sizes

Question

I'm relatively new to the world of recurrent neural nets and I'm trying to build a classifier using an LSTM model to predict HIV activity from a given molecule (the original dataset can be found here ).

I have sequences of different lengths (from few dozen to almost 400 characters) but I'm not sure how to proceed. Let's say that I have a dataset structured like so:

import pandas as pd
import random
import string

random.seed(42)
seqs = [[random.choice(string.ascii_letters) for i in range(random.randint(1,10))] for i in range(5)]
classes = [random.randint(0,1) for i in range(5)]
df = pd.DataFrame({
    "seq": seqs,
    "class": classes
})
print(df)

    seq                             class
0   [b, V]                          0
1   [p, o, i, V, g]                 0
2   [f, L, B, c, b, f, n, o, G]     1
3   [b, J, m, T, P, S, I, A, o, C]  0
4   [r, Z, a, W, Z, k, S, B, v, r]  0

I know I should:

• one hot encode the elements,
• perform masking and padding

But I don't know how to perform it in Keras/TF2 and I can't find any resources online that explain how to code something similar.

Answer 1

masking and padding are indeed popular preprocessing choices when working with LSTM. The easiest way is to pad all the short sequences to the same length as the longest sequence.

Here is an example of masking and padding in Keras

https://www.tensorflow.org/guide/keras/masking_and_padding

Is that what you're looking for?

raw_inputs = [
    [711, 632, 71],
    [73, 8, 3215, 55, 927],
    [83, 91, 1, 645, 1253, 927],
]

# By default, this will pad using 0s; it is configurable via the
# "value" parameter.
# Note that you could "pre" padding (at the beginning) or
# "post" padding (at the end).
# We recommend using "post" padding when working with RNN layers
# (in order to be able to use the
# CuDNN implementation of the layers).
padded_inputs = tf.keras.preprocessing.sequence.pad_sequences(
    raw_inputs, padding="post"
)
print(padded_inputs)

Output:

[[ 711  632   71    0    0    0]
 [  73    8 3215   55  927    0]
 [  83   91    1  645 1253  927]]

Besides, you can also add start and end tokens to your sequences, like this example (https://www.tensorflow.org/tutorials/text/nmt_with_attention)

def preprocess_sentence(w):
  w = unicode_to_ascii(w.lower().strip())

  # creating a space between a word and the punctuation following it
  # eg: "he is a boy." => "he is a boy ."
  # Reference:- https://stackoverflow.com/questions/3645931/python-padding-punctuation-with-white-spaces-keeping-punctuation
  w = re.sub(r"([?.!,¿])", r" \1 ", w)
  w = re.sub(r'[" "]+', " ", w)

  # replacing everything with space except (a-z, A-Z, ".", "?", "!", ",")
  w = re.sub(r"[^a-zA-Z?.!,¿]+", " ", w)

  w = w.strip()

  # adding a start and an end token to the sentence
  # so that the model know when to start and stop predicting.
  w = '<start> ' + w + ' <end>'
  return w

```