0
$\begingroup$

The Problem:

I am very new to TF and Keras. I am attempting to train a time-series LSTM. When using only a few parameters as a test, the model seems to work fine. Once I increase the parameters to the full 17010, I almost instantly receive a loss: nan.

Network Specs:

My inputs have been normalized into a range between 0 and 1 and almost all look fairly clean with no significant outliers. When graphing each of the parameters, they look similar to this:

enter image description here

Some specs on my network:

  • Training examples (Time steps): 9835
  • Input parameters: 17010
  • Input sequence length: 30
  • Output parameters: 1
  • Output sequence length: 1

In summary, the entire sequence of inputs coming from the Timeseries generator end up with shape (30, 17010, 9835) I have tried batch sizes between 32 and 256.

Note: My inputs have a lot of missing values. Since my model is attempting to predict the increase or decrease as percent change from the previous time step (1 day). ie. y[i] > 0.5 --> increase and y[i] < 0.5 --> decrease. To fill these missing values I have assigned them a value of 0.5

I Have Tried:

  1. Decreasing dropout rate
  2. Clipping gradients (don't think this is necessary for me since everything is between 0 and 1)
  3. Checked all inputs for NaN or Inf values
  4. Increasing batch size
  5. Decreasing my learning rate
  6. Reducing number of input parameters to 3406

I have found conflicting information on the number of layers and units that are appropriate but I have attempted to play with these values a little bit. I have tried between 32 and 2000 hidden units.

My Code:

I am using a keras.sequence.TimeseriesGenerator for both training and validation like this:

dataset_train = keras.preprocessing.sequence.TimeseriesGenerator(
    x_train,
    y_train,
    length=past,
    batch_size=batch_size,
)
model = Sequential()
model.add(LSTM(hidden_units, return_sequences=False, activation='relu', input_shape=(past, dataset_train.data.shape[1])))
model.add(Dropout(0.2))
model.add(Dense(hidden_units))
model.compile(optimizer=keras.optimizers.Adam(learning_rate=learning_rate), loss="mse")

path_checkpoint = "model_checkpoint.h5"
es_callback = keras.callbacks.EarlyStopping(monitor="val_loss", min_delta=0, patience=5)
model.summary()
modelckpt_callback = keras.callbacks.ModelCheckpoint(
    monitor="val_loss",
    filepath=path_checkpoint,
    verbose=1,
    save_weights_only=True,
    save_best_only=True,
)

history = model.fit(
    dataset_train,
    epochs=epochs,
    validation_data=dataset_val,
    callbacks=[es_callback, modelckpt_callback, PlotLossesKeras()],
)

I'm not sure if this many parameters are near impossible to train with since I have a smaller number of examples or if something else is causing this. I appreciate any advice you can give me.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.