The Problem:

I am very new to TF and Keras. I am attempting to train a time-series LSTM. When using only a few parameters as a test, the model seems to work fine. Once I increase the parameters to the full 17010, I almost instantly receive a loss: nan.

Network Specs:

My inputs have been normalized into a range between 0 and 1 and almost all look fairly clean with no significant outliers. When graphing each of the parameters, they look similar to this:

enter image description here

Some specs on my network:

  • Training examples (Time steps): 9835
  • Input parameters: 17010
  • Input sequence length: 30
  • Output parameters: 1
  • Output sequence length: 1

In summary, the entire sequence of inputs coming from the Timeseries generator end up with shape (30, 17010, 9835) I have tried batch sizes between 32 and 256.

Note: My inputs have a lot of missing values. Since my model is attempting to predict the increase or decrease as percent change from the previous time step (1 day). ie. y[i] > 0.5 --> increase and y[i] < 0.5 --> decrease. To fill these missing values I have assigned them a value of 0.5

I Have Tried:

  1. Decreasing dropout rate
  2. Clipping gradients (don't think this is necessary for me since everything is between 0 and 1)
  3. Checked all inputs for NaN or Inf values
  4. Increasing batch size
  5. Decreasing my learning rate
  6. Reducing number of input parameters to 3406

I have found conflicting information on the number of layers and units that are appropriate but I have attempted to play with these values a little bit. I have tried between 32 and 2000 hidden units.

My Code:

I am using a keras.sequence.TimeseriesGenerator for both training and validation like this:

dataset_train = keras.preprocessing.sequence.TimeseriesGenerator(
model = Sequential()
model.add(LSTM(hidden_units, return_sequences=False, activation='relu', input_shape=(past, dataset_train.data.shape[1])))
model.compile(optimizer=keras.optimizers.Adam(learning_rate=learning_rate), loss="mse")

path_checkpoint = "model_checkpoint.h5"
es_callback = keras.callbacks.EarlyStopping(monitor="val_loss", min_delta=0, patience=5)
modelckpt_callback = keras.callbacks.ModelCheckpoint(

history = model.fit(
    callbacks=[es_callback, modelckpt_callback, PlotLossesKeras()],

I'm not sure if this many parameters are near impossible to train with since I have a smaller number of examples or if something else is causing this. I appreciate any advice you can give me.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.