Use LSTM to predict the proportion of steps with nonzero feature values

Question

I am trying to do a simple regression for sequences. Each input $X_i$ is a $n=2000$ by 1 matrix, formatted as $n_i$ 0-s followed by $(n-n_i)$ 1-s. The output $y_i$ should be $n_i/n$, i.e. the proportion of steps with nonzero feature values.

The code for generating the data

ns = [1087, 280, 1013, 300, 311, 260, 960, 281, 417, 311, 102, 251, 748, 428, 802, 512, 61, 80, 1700, 693, 147, 577, 280, 196, 377, 281, 88, 359, 250, 330, 679, 201, 284, 1129, 151, 285, 722, 1014, 153, 202, 256, 99, 661, 415, 134, 269, 196, 137, 436, 617]
Xs = np.array([[[0] for i in range(2000-x)]+[[1] for i in range(x)] for x in ns])
Ys = np.mean(Xs,axis=(1,2))
trainX, testX, trainY, testY = Xs[:45], Xs[45:], Ys[:45], Ys[45:]

I tried to use LSTM and an output layer to do the prediction, i.e.

model = Sequential()
model.add(Masking(mask_value=0.00, input_shape=(trainX.shape[1],trainX.shape[2])))
model.add((LSTM(10,activation='tanh',unit_forget_bias=True,input_shape=(trainX.shape[1],trainX.shape[2]))))
model.add(Dense(1,activation='relu'))
model.compile(optimizer=adam(),loss='mse',metrics=['mae'],)
model.fit(trainX,trainY,epochs=50,validation_data=(testX,testY),verbose=1)

Yet I found the model fails to predict what I want, instead it predicts every sequence with almost the same $y$ value. Just wondering if there's any solution to solve this?

Answer 1

First, you have too little data. 50 samples. Think of it. Of all possible 2001 sequences of that kind, you are only feeding 50 (less than 2.5%). Your question is actually a very good case to show the importance of big data for training a neural network.

And second, it is the problem stated in this arxiv article made by Uber empolyees - even a deep neural network with nonlinear units often cannot generalize to solution of a simple linear problem.

Your problem still can be approximately solved with

1. Reducing batch size

It is well known that increasing batch size leads to faster but poorer convergence. Particularly in your case, you should use line:

model.fit(trainX,trainY,epochs=50,validation_data=(testX,testY),verbose=1,
    batch_size=1,
    shuffle=True)

instead of

model.fit(trainX,trainY,epochs=50,validation_data=(testX,testY),verbose=1)

even knowing that it will be slooooow...

2. Modifying LSTM layer: removing unit_forget_bias=True and adding orthogonal initializer with small gain - explained in this article:

from keras.initializers import orthogonal
model.add(LSTM(10, activation='tanh', #unit_forget_bias=True,
    kernel_initializer=orthogonal(gain=.01),
))

3. Increasing coverage of the space of all possible samples

In the code below SEQ_LEN=2000 and N_SAMPLES=50 is your case. Here we generate random sequences like ones that your code above (although not the same sequences as yours). Now, try experimenting with values SEQ_LEN=200 and N_SAMPLES=80 and then SEQ_LEN=200 and N_SAMPLES=130. See if with N_SAMPLES<SEQ_LEN/2 your net converges at all, while in second case it will!

import numpy as np

SEQ_LEN = 2000
N_SAMPLES = 50
n_test_samples = int(.1*N_SAMPLES)
if N_SAMPLES>SEQ_LEN+1: raise ValueError("Can't create more samples of this type than SEQ_LEN+1 is.")
ns = np.random.randint(0,SEQ_LEN,size=(N_SAMPLES,))
all_Xs = np.triu ( np.ones(shape=(SEQ_LEN+1,SEQ_LEN)) )
np.random.shuffle(all_Xs)

Xs = all_Xs[:N_SAMPLES, ...]
Ys = Xs.mean(axis=1)
Xs = Xs.reshape(N_SAMPLES, SEQ_LEN, -1)
trainX, testX, trainY, testY = Xs[:-n_test_samples], Xs[-n_test_samples:], Ys[:-n_test_samples], Ys[-n_test_samples:]

import tensorflow # To prevent
import tensorflow as tf # to declarations to prevent cryptic errors on my Windows laptop

from keras import Sequential
from keras.layers import Masking, LSTM, Dense
from keras.optimizers import adam
from keras.initializers import orthogonal
from keras.layers import Flatten

model = Sequential()
model.add(Masking(mask_value=0.00, input_shape=(trainX.shape[1],trainX.shape[2])))
model.add(LSTM(10, activation='tanh', #unit_forget_bias=True,
    input_shape=(trainX.shape[1],trainX.shape[2]),
    kernel_initializer=orthogonal(gain=.01),
))
model.add(Dense(1,activation='relu'))
model.compile(optimizer=adam(),loss='mse',metrics=['mae'])
model.fit(trainX,trainY,epochs=50,validation_data=(testX,testY),verbose=1,
    batch_size=1,
    shuffle=True)

testY_PRED = model.predict(testX)
print (np.concatenate(( testY_PRED, testY.reshape(-1,1)), axis=1) )

I haven also tested case with 300 samples and sequence length of 500 (converges). However, it is rather long to test if network converges with SEQ_LEN=2000 and let's say N_SAMPES=1200 (don't hope it will with N_SAMPLES=50) - it is up to you :)