3
$\begingroup$

I'm experimenting (read: playing around) with LSTMs on Keras. I want to train an LSTM network so it would "track" sine waves, that is, given sine waves with different wave length, phases and lengths, it would output the rest of wave. In a sense, a many-to-many problem.

I decided that the network would need to "track" 5 sine waves simultaneously. I would generate random sine waves and use the 50 last data points as the required output of the NN.

My code is a follows:

from keras.models import Sequential
from keras.layers import LSTM, Dense, TimeDistributed,Lambda, Dropout
import numpy as np

model = Sequential()

model.add(LSTM(32, return_sequences=True, input_shape=(None, 5)))
model.add(Dropout(0.2))
model.add(LSTM(32, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(32, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(32, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(32, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(5, return_sequences=True))
model.add(Lambda(lambda x: x[:, -50:, :])) # Grab only the last 50 values



print(model.summary())

model.compile(loss='mean_squared_error',
              optimizer='adam')


def make_line(length):
    shift= np.random.random()
    wavelength = 5+10*np.random.random()
    a=np.arange(length)
    answer=np.sin(a/wavelength+shift)
    return answer




def make_data(seq_num,seq_len,dim):
    data=np.array([]).reshape(0,seq_len,dim)
    for i in range (seq_num):
        mini_data=np.array([]).reshape(0,seq_len)
        for j in range (dim):
            line = make_line(seq_len)
            line=line.reshape(1,seq_len)            
            mini_data=np.append(mini_data,line,axis=0)
        mini_data=np.swapaxes(mini_data,1,0)
        mini_data=mini_data.reshape(1,seq_len,dim)      
        data=np.append(data,mini_data,axis=0)
    return (data)



def train_generator():
    while True:
        sequence_length = np.random.randint(50, 150)+50     
        data=make_data(1000,sequence_length,5)
        x_train = data[:,:-50,:] # all but last 50      
        y_train = (data[:, -50:, :]) # last 50      
        yield x_train, y_train


model.fit_generator(train_generator(), steps_per_epoch=30, epochs=100, verbose=1,validation_data=train_generator(),validation_steps=30)
model.save('vi2h_model.h5')

I then try to generate data and use the model to predict the continuation of a sine wave:

def data_generator():
    while True:
        sequence_length = np.random.randint(50, 150)+50
        data=make_data(1000,sequence_length,5)
        x_train = data[:,:-50,:]    
        y_train = (data[:, -50:, :])                
        return x_train, y_train


x,y= data_generator()   

model = load_model('vi2h_model.h5')
pred=model.predict (x)
np.save ('pred.npy',pred)
np.save ('y.npy',y)
np.save ('x.npy',x)

When inspecting the ability of the network to predict the continuation of a sine wave (comparing y[0,:,0] to pred[0,:,0] for example, in order to compare the first sine wave of the first generated datum), see that my network performed very badly:

The predicted vs actual 50 last points in a sample sine wave, placed after the part of the wave used as input for the NN

(Edit: legend: The predicted vs actual 50 last points in a sample sine wave (blue and orange, respectively), placed after the part of the wave used as input for the NN)

What did I do wrong? And how should I my code to correctly track sine waves? Many thanks!

$\endgroup$
  • 1
    $\begingroup$ How long did you train it for? Did you wait until your validation accuracy started to decrease? What are the default activations for LSTM? $\endgroup$ – kbrose May 22 '18 at 13:38
  • $\begingroup$ I trained it for 100 epochs; Both validation and train loss got "stuck" so to speak after this period (given the task I do not have an accuracy statistic). Default LSTM activations were used, that is, tanh for the "data pathway" with sigmoids controlling the gates. $\endgroup$ – Lafayette May 22 '18 at 13:53
  • $\begingroup$ Your problem may be having tanh in your output layer. You’re restricting the output range that way. I suggest linear (no) activation in final layer. $\endgroup$ – kbrose May 22 '18 at 17:08
  • $\begingroup$ Yes, I have replaced that with a linear activation, as well as replacing the last layer with a time-distributed dense layer - as can be seen below. $\endgroup$ – Lafayette May 23 '18 at 7:57
3
$\begingroup$

I have added a dense layer to the network:

model = Sequential()

model.add(LSTM(32, return_sequences=True, input_shape=(None, 5)))
model.add(Dropout(0.2))
model.add(LSTM(32, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(32, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(32, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(32, return_sequences=True))
model.add(Dropout(0.2))
model.add(TimeDistributed(Dense(5)))  # <--- This is new :)
model.add(Activation('linear')) 
model.add(Lambda(lambda x: x[:, -50:, :])) # Grab only the last 50 values

This has provided some improvement :

enter image description here

These are two sine waves and the predictions of the last 50 data points of them. As can be seen, the results have improved, with some of them being okay approximations of the sine waves. Other times... not so much.

Still a lot better than the previous results.

I then tried another approach: Instead of having one (multilayered) LSTM, in which I ignore all but the 50 inputs (in a sense trying to predict from each one of the 50 last values the value located 50 places after it... which might not be an ideal approach) I split the network into 2 LSTMs, with one receiving the input and "injecting" its results via a dense layer into the second LTSM layer which generates the output.

features_num=5 

model.add(LSTM(40, return_sequences=True, input_shape=(None, 5)))
model.add(LSTM(40, return_sequences=False))

model.add(Dense(80))
model.add(Activation('tanh')) 
model.add(RepeatVector(50)) # 50 output points

model.add(LSTM(40, return_sequences=True))
model.add(LSTM(40, return_sequences=True))

model.add(TimeDistributed(Dense(features_num)))
model.add(Activation('linear')) 

Results are, for the most part, better: enter image description here

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.