# How to deal with a constant value as an output from neural network?

I am using feedforward neural network for regression and what I get as a result of prediction is a constant value visible on the graph below:

Data I use are typical standardised tabular numbers. The architecture is as follows:

model.add(Dense(units=512, activation='relu', input_shape=x_train.shape[1],)))

reduce_lr = ReduceLROnPlateau(
monitor='val_loss',
factor=0.9,
patience=10,
min_lr=0.0001,
verbose=1)

tensorboard = TensorBoard(log_dir="logs\{}".format(NAME))

history = model.fit(
x_train,
y_train,
epochs=500,
verbose=10,
batch_size=128,
callbacks=[reduce_lr, tensorboard],
validation_split=0.1)


It seems for me that all weights are zeroed and only constant bias is present here, since for different data samples from a test set I get the same value.

I understand that the algorithm has found smallest MSE for such a constant value, but is there a way of avoiding such situation, since straight line is not really good solution for my project?

[EDIT] Adding learning curves for trainging and validation sets.

• Could you share plots of training and test loss? Or say, does the model learn training examples? – Lana Aug 26 '19 at 16:50
• Yes, I edited the question. – Makintosz Aug 26 '19 at 17:32
• So this is not an LSTM model? Since your data looks like classical time series, LSTM would be the first thing to try, IMO. Your regress towards the mean as it appears. – Peter Aug 26 '19 at 17:37
• It is not LSTM, however model has some lagged features. I used LSTMs for similar data and eventually ended up with same problem. However before that I have not noticed any big difference in the results. – Makintosz Aug 26 '19 at 17:47

adam = optimizers.Adam(lr=0.005)