The Problem:

I am very new to TF and Keras. I am attempting to train a time-series LSTM. When using only a few parameters as a test, the model seems to work fine. Once I increase the parameters to the full 17010, I almost instantly receive a loss: nan.

Network Specs:

My inputs have been normalized into a range between 0 and 1 and almost all look fairly clean with no significant outliers. When graphing each of the parameters, they look similar to this:

Some specs on my network:

• Training examples (Time steps): 9835
• Input parameters: 17010
• Input sequence length: 30
• Output parameters: 1
• Output sequence length: 1

In summary, the entire sequence of inputs coming from the Timeseries generator end up with shape (30, 17010, 9835) I have tried batch sizes between 32 and 256.

Note: My inputs have a lot of missing values. Since my model is attempting to predict the increase or decrease as percent change from the previous time step (1 day). ie. y[i] > 0.5 --> increase and y[i] < 0.5 --> decrease. To fill these missing values I have assigned them a value of 0.5

I Have Tried:

1. Decreasing dropout rate
2. Clipping gradients (don't think this is necessary for me since everything is between 0 and 1)
3. Checked all inputs for NaN or Inf values
4. Increasing batch size
5. Decreasing my learning rate
6. Reducing number of input parameters to 3406

I have found conflicting information on the number of layers and units that are appropriate but I have attempted to play with these values a little bit. I have tried between 32 and 2000 hidden units.

My Code:

I am using a keras.sequence.TimeseriesGenerator for both training and validation like this:

dataset_train = keras.preprocessing.sequence.TimeseriesGenerator(
x_train,
y_train,
length=past,
batch_size=batch_size,
)

model = Sequential()

path_checkpoint = "model_checkpoint.h5"
es_callback = keras.callbacks.EarlyStopping(monitor="val_loss", min_delta=0, patience=5)
model.summary()
modelckpt_callback = keras.callbacks.ModelCheckpoint(
monitor="val_loss",
filepath=path_checkpoint,
verbose=1,
save_weights_only=True,
save_best_only=True,
)

history = model.fit(
dataset_train,
epochs=epochs,
validation_data=dataset_val,
callbacks=[es_callback, modelckpt_callback, PlotLossesKeras()],
)


I'm not sure if this many parameters are near impossible to train with since I have a smaller number of examples or if something else is causing this. I appreciate any advice you can give me.