I found PhasedLSTM inspirational, and used it (PLSTM: Phased LSTM in Keras) to perform the regression (to find the correlation between an input sequence and an output sequence), with Adam optimizer
model.compile(loss="mean_squared_error", sample_weight_mode="temporal", optimizer = keras.optimizers.Adam(lr=0.01, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0))
, then I got "nan" in the weight , also the loss in the beginning of the first epoch, even I used gradient clipping.
The generated data for training (x1 - dashed blue as input in low freuency, y - orange as output in higher frequency): (little modified from Extract weights from Keras LSTM)
After checked the weights in PLSTM layer, I found the values of timegate-kernel getting larger and larger, then the weights get to "nan". (The first two rows)
I changed to standard LSTM (other settings and learning rate [still 0.01] the same), the loss converges. Therefore, I traced the source code of PLSTM, considering the initialization of timegate_kernel matters, but have limited progress.
However, when changing the learning rate from 0.01 to 0.001, I got not-too-bad results. Here is the prediction from LSTM (learning rate = 0.001, mse, epoch = 200, batch_size = 32, Dense in final layer under tanh)
I read the paper Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences, and knows PhasedLSTM is for event-based long sequences, which is not my case here. But I am still wondering why the weights in timegate_kernel getting larger and larger.
Does anyone has the similar issue? Any suggestion is appreciated. The relevant code is at github: PhasedLSTM regression