0
$\begingroup$

I have a time series prediction problem from building an LSTM.

My code:

def create_model():
    model = Sequential()
    model.add(LSTM(50,kernel_regularizer=l2(0.01), recurrent_regularizer=l2(0.01), bias_regularizer=l2(0.01), input_shape=(train_X.shape[1], train_X.shape[2])))
    model.add(Dropout(0.591))
    model.add(Dense(1))
    model.compile(loss='mean_squared_error', optimizer='adam')
    return model

Then I train the model on 5 splits:

tss = TimeSeriesSplit(n_splits = 5)
X = data.drop(labels=['target_prediction'], axis=1)
y = data['target_prediction'] 
for train_index, test_index in tss.split(X):
   train_X, test_X = X.iloc[train_index, :].values, X.iloc[test_index,:].values
   train_y, test_y = y.iloc[train_index].values, y.iloc[test_index].values
   model=create_model()
   history = model.fit(train_X, train_y, epochs=10, batch_size=64,validation_data=(test_X, test_y), verbose=0, shuffle=False)

I then get an overfitting problem. The graph of the loss is attached enter image description here

I am not sure why this is overfitting when I use regularizers in my Keras model. Any help is appreciated.

EDIT: I tried the architectures:

def create_model():
    model = Sequential()
    model.add(LSTM(20, input_shape=(train_X.shape[1], train_X.shape[2])))
    model.add(Dense(1))
    model.compile(loss='mean_squared_error', optimizer='adam')
    return model


def create_model(x,y):
    # define LSTM
    model = Sequential()
    model.add(Bidirectional(LSTM(20, return_sequences=True), input_shape=(x,y)))
    model.add(TimeDistributed(Dense(1, activation='sigmoid')))
    model.compile(loss='mean_squared_error', optimizer='adam')
    return model 

However, the model is still overfitting.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.