0
$\begingroup$

I am using feedforward neural network for regression and what I get as a result of prediction is a constant value visible on the graph below:output from my NN

Data I use are typical standardised tabular numbers. The architecture is as follows:

model.add(Dense(units=512, activation='relu', input_shape=x_train.shape[1],)))
model.add(Dropout(0.2))
model.add(Dense(units=512, activation='relu'))
model.add(Dropout(0.3))
model.add(Dense(units=256, activation='relu'))
model.add(Dropout(0.3))
model.add(Dense(units=128, activation='relu'))
model.add(Dense(units=128, activation='relu'))
model.add(Dense(units=1))

adam = optimizers.Adam(lr=0.1)

model.compile(loss='mean_squared_error', optimizer=adam)

reduce_lr = ReduceLROnPlateau(
    monitor='val_loss',
    factor=0.9,
    patience=10,
    min_lr=0.0001,
    verbose=1)

tensorboard = TensorBoard(log_dir="logs\{}".format(NAME))

history = model.fit(
    x_train,
    y_train,
    epochs=500,
    verbose=10,
    batch_size=128,
    callbacks=[reduce_lr, tensorboard],
    validation_split=0.1)

It seems for me that all weights are zeroed and only constant bias is present here, since for different data samples from a test set I get the same value.

I understand that the algorithm has found smallest MSE for such a constant value, but is there a way of avoiding such situation, since straight line is not really good solution for my project?

[EDIT] Adding learning curves for trainging and validation sets. enter image description here

$\endgroup$
  • $\begingroup$ Could you share plots of training and test loss? Or say, does the model learn training examples? $\endgroup$ – Lana Aug 26 at 16:50
  • $\begingroup$ Yes, I edited the question. $\endgroup$ – Makintosz Aug 26 at 17:32
  • $\begingroup$ So this is not an LSTM model? Since your data looks like classical time series, LSTM would be the first thing to try, IMO. Your regress towards the mean as it appears. $\endgroup$ – Peter Aug 26 at 17:37
  • $\begingroup$ It is not LSTM, however model has some lagged features. I used LSTMs for similar data and eventually ended up with same problem. However before that I have not noticed any big difference in the results. $\endgroup$ – Makintosz Aug 26 at 17:47
0
$\begingroup$

Lowering the initial learning rate from 0.1 to 0.005 helped the problem.

adam = optimizers.Adam(lr=0.005)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.