I am very new to TF and Keras. I am attempting to train a time-series LSTM. When using only a few parameters as a test, the model seems to work fine. Once I increase the parameters to the full 17010, I almost instantly receive a
My inputs have been normalized into a range between 0 and 1 and almost all look fairly clean with no significant outliers. When graphing each of the parameters, they look similar to this:
Some specs on my network:
- Training examples (Time steps): 9835
- Input parameters: 17010
- Input sequence length: 30
- Output parameters: 1
- Output sequence length: 1
In summary, the entire sequence of inputs coming from the Timeseries generator end up with shape (30, 17010, 9835) I have tried batch sizes between 32 and 256.
Note: My inputs have a lot of missing values. Since my model is attempting to predict the increase or decrease as percent change from the previous time step (1 day). ie.
y[i] > 0.5 --> increase and
y[i] < 0.5 --> decrease. To fill these missing values I have assigned them a value of
I Have Tried:
- Decreasing dropout rate
- Clipping gradients (don't think this is necessary for me since everything is between 0 and 1)
- Checked all inputs for NaN or Inf values
- Increasing batch size
- Decreasing my learning rate
- Reducing number of input parameters to 3406
I have found conflicting information on the number of layers and units that are appropriate but I have attempted to play with these values a little bit. I have tried between 32 and 2000 hidden units.
I am using a
keras.sequence.TimeseriesGenerator for both training and validation like this:
dataset_train = keras.preprocessing.sequence.TimeseriesGenerator( x_train, y_train, length=past, batch_size=batch_size, )
model = Sequential() model.add(LSTM(hidden_units, return_sequences=False, activation='relu', input_shape=(past, dataset_train.data.shape))) model.add(Dropout(0.2)) model.add(Dense(hidden_units)) model.compile(optimizer=keras.optimizers.Adam(learning_rate=learning_rate), loss="mse") path_checkpoint = "model_checkpoint.h5" es_callback = keras.callbacks.EarlyStopping(monitor="val_loss", min_delta=0, patience=5) model.summary() modelckpt_callback = keras.callbacks.ModelCheckpoint( monitor="val_loss", filepath=path_checkpoint, verbose=1, save_weights_only=True, save_best_only=True, ) history = model.fit( dataset_train, epochs=epochs, validation_data=dataset_val, callbacks=[es_callback, modelckpt_callback, PlotLossesKeras()], )
I'm not sure if this many parameters are near impossible to train with since I have a smaller number of examples or if something else is causing this. I appreciate any advice you can give me.