0
$\begingroup$

I found PhasedLSTM inspirational, and used it (PLSTM: Phased LSTM in Keras) to perform the regression (to find the correlation between an input sequence and an output sequence), with Adam optimizer

model.compile(loss="mean_squared_error", sample_weight_mode="temporal", optimizer = keras.optimizers.Adam(lr=0.01, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0))

, then I got "nan" in the weight , also the loss in the beginning of the first epoch, even I used gradient clipping.

The generated data for training (x1 - dashed blue as input in low freuency, y - orange as output in higher frequency): (little modified from Extract weights from Keras LSTM)

enter image description here

After checked the weights in PLSTM layer, I found the values of timegate-kernel getting larger and larger, then the weights get to "nan". (The first two rows)

enter image description here

I changed to standard LSTM (other settings and learning rate [still 0.01] the same), the loss converges. Therefore, I traced the source code of PLSTM, considering the initialization of timegate_kernel matters, but have limited progress.

However, when changing the learning rate from 0.01 to 0.001, I got not-too-bad results. Here is the prediction from LSTM (learning rate = 0.001, mse, epoch = 200, batch_size = 32, Dense in final layer under tanh) enter image description here

Here is the prediction from PhasedLSTM [the same setting as LSTM] (learning rate = 0.001, mse, epoch = 200, batch_size = 32, Dense in final layer under tanh) enter image description here

I read the paper Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences, and knows PhasedLSTM is for event-based long sequences, which is not my case here. But I am still wondering why the weights in timegate_kernel getting larger and larger.

Does anyone has the similar issue? Any suggestion is appreciated. The relevant code is at github: PhasedLSTM regression

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.