# no decrease loss and val_loss

I try to train a neural network for time series. I use some data from Covid, mainly the goal is knowing 14 days of number of people at hospital to predict the number at J+1. I have use some early stopping to not over fit, but almost one time over two the learning stop at patience+1 and there is no decrease of loss and val_loss. I have tries to move hyperparameters like learning rate but the problem is always here. Any guess?

The main code is below and the whole code with data :https://github.com/paullaurain/prediction

import os
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
from keras.models import Sequential
from keras.layers import Dense
from keras.callbacks import EarlyStopping
from keras.callbacks import ModelCheckpoint
from sklearn.preprocessing import MinMaxScaler

# fit a model
def model_fit(data, config):
# unpack config
n_in,n_out, n_nodes, n_epochs, n_batch,p,pl = config
# prepare data
DATA = series_to_supervised(data, n_in, n_out)
X, Y = DATA[:, :-n_out], DATA[:, n_in:]
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.1)
# define model
model = Sequential()
# fit
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=p)
file='best_modelDense.hdf5'
mc = ModelCheckpoint(filepath=file, monitor='loss', mode='min', verbose=0, save_best_only=True)
history=model.fit(X_train, y_train, validation_data=(X_test,y_test), epochs=n_epochs, verbose=0,batch_size=n_batch, callbacks=[es,mc])
if pl:
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()
os.remove(file)
return history.history['val_loss'][-p], saved_model

# repeat evaluation of a config
def repeat_evaluate(data,n_test, config, n_repeat,plot):
# rescale data
scaler = MinMaxScaler(feature_range=(0, 1))
scaler = scaler.fit(data)
scaled_data=scaler.transform(data)
scores=[]
for _ in range(n_repeat):
score, model= model_fit(scaled_data[:-n_test], config)
scores.append(score)
# plot the prediction id asked
if plot:
y=[]
x=[]
for i in range(n_test,0,-1):
y.append(float(model.predict(scaled_data[-14-i:-i].reshape(1,14))))
x.append(scaled_data[-i])
X=scaler.inverse_transform(x)
plt.plot(X)
Y=scaler.inverse_transform(np.array([y]))
plt.plot(Y.reshape(10,1))
plt.title('result')
plt.legend(['real', 'prdiction'], loc='upper left')

plt.show()
return scores

# summarize model performance
def summarize_scores(name, scores):
# print a summary
scores_m, score_std = mean(scores), std(scores)
print( '%s: %.3f RMSE (+/- %.3f)' % (name, scores_m, score_std))
# box and whisker plot
pyplot.boxplot(scores)
pyplot.show()

#setting variable
n_in=14
n_out=1
n_repeat=5
n_test=10
# define config n_in, n_out, n_nodes, n_epochs, n_batch, pateince, draw loss
config = [n_in, n_out, 10, 2000, 50, 200,True]
# compute scores
scores = repeat_evaluate(data,n_test, config, n_repeat, True)
print(scores)
# summarize scores
summarize_scores('mlp ', scores)


Result:

• You should ask this on Data Science Stack Exchange. I suggested the question to be migrated. – Romain Reboulleau Nov 9 '20 at 19:54
• @RomainReboulleau I agree, the problem is not caused by the code, I guess using a shallower model will resolve the problem. – meTchaikovsky Nov 11 '20 at 0:59
• I am agree to migrate the question, how to do it preserving the bounty? – Paul Nov 11 '20 at 10:43
• I have run your model and found no such problems in losses. The losses decrease perfectly. As @meTchaikovsky said in the answer, probably the problem is due to model initialization. Also I have used tensorflow.keras instead of keras – hafiz031 Nov 11 '20 at 12:52
• Use LeaklyReLU instead of ReLU and the problem will be fixed. Simply remove activation="relu" from Dense() and add another layer of LeaklyReLU after each of the Dense layers like: model.add(LeakyReLU(alpha=0.05)). I ran your code with this change for 100 times (n_repeat=100) and this problem didn't occur for a single time. – hafiz031 Nov 12 '20 at 12:33

Please have a look at your weights after training. I assume your Neurons die due to relu activation as they output Zero for Input < 0.

Unfortunately, the ReLU activation function is not perfect. It suffers from a problem known as the dying ReLUs: during training, some neurons effectively “die,” meaning they stop outputting anything other than 0. In some cases, you may find that half of your network’s neurons are dead, especially if you used a large learning rate. A neuron dies when its weights get tweaked in such a way that the weighted sum of its inputs are negative for all instances in the training set. When this happens, it just keeps outputting zeros, and Gradient Descent does not affect it anymore because the gradient of the ReLU function is zero when its input is negative.

From Hands-on Machine Learning with Scikit-Learn, Keras & TensorFlow Concepts, Tools, and Techniques to Build Intelligent Systems, Aurélien Géron, 2019

So to combat this problem remove the ReLU activations and use LeaklyReLU instead. So for your case following are the changes:

from tensorflow.keras.layers import LeakyReLU # for leakly relu



After changing your code as specified the problem should be fixed.

But in general I'm with @meTchaikovsky, for time series data recurrent neural networks are better suited for modelling.

• Indeed, replacing 'relu' by 'tanh' (for instance) solves the issue!! But since I am doing a regression, it seems that 'tanh' is not really appropriate since I am expected positive value, what do you advice. 'tanh' only on hidden layer and 'relu' on outpout, the converse, other activation fonctions? – Paul Nov 12 '20 at 12:58
• For the last/output layer you can stick to ReLU, use sigmoid or simply no activation at all. – Andre S. Nov 12 '20 at 15:19
• @AndreS. it is a regression problem (see, mse is used as the loss function). So sigmoid is not an appropriate activation function in this case. sigmoid is for binary classification. – hafiz031 Nov 12 '20 at 15:53
• @hafiz031 you can still use it for regression, its output will only be bounded within (0;1), but it predicts positive values. – Andre S. Nov 12 '20 at 16:03
• @Paul accuracy also depends on the initialization of weights and biases of the network. Here is a nice post (wandb.com/articles/…) on this. You may find it useful. Cheers! – hafiz031 Nov 12 '20 at 20:37

Because the data is time series while only Dense layers are used in the model, the problem is caused by model initialization. A model with a 'bad' initialization will constantly predict zero, as you will see by running the script below.

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler
from tensorflow.keras.models import Model,Sequential
from tensorflow.keras.layers import Dense

import numpy as np
import tensorflow as tf

# fix keras random state
# https://stackoverflow.com/a/52897216/8366805
seed_val = 94
np.random.seed(seed_val)
tf.set_random_seed(seed_val)
# Configure a new global tensorflow session
sess = tf.compat.v1.Session(graph=tf.compat.v1.get_default_graph(), config=session_conf)
tf.compat.v1.keras.backend.set_session(sess)

# main
def series_to_supervised(data,n_in,n_out):

df = pd.DataFrame(data)
cols = list()
for i in range(n_in,0,-1): cols.append(df.shift(i))
for i in range(0, n_out): cols.append(df.shift(-i))
agg = pd.concat(cols,axis=1)
agg.dropna(inplace=True)

return agg.values

n_in,n_out = 14,1
scaler = MinMaxScaler(feature_range=(0, 1))
scaler = scaler.fit(data)
scaled_data=scaler.transform(data)
DATA = series_to_supervised(scaled_data[:-10], n_in, n_out)
X, Y = DATA[:, :-n_out], DATA[:, n_in:]
X_train,X_test,y_train,y_test = train_test_split(X,Y,test_size=0.1,random_state=49)

model = Sequential()
n_nodes = 10
# fit
history = model.fit(X_train,y_train,validation_data=(X_test,y_test),epochs=20,)

pred = model.predict(X,)
print('model prediction, mean %.3f, std %.3f' % (np.mean(pred),np.std(pred)))

for ind in range(3):
intermediate_layer_model = Model(inputs=model.input,outputs=model.get_layer('dense_%i' % ind).output)
pred = intermediate_layer_model.predict(X)
print('layer %i, mean %.3f, std %.3f, min %.3f, max %.3f' % (ind,np.mean(pred),np.std(pred),np.min(pred),np.max(pred)))


In this script, I saved the array data in OP's post, to data.npy, which can be found in this repo, for simplicity. Besides, I fixed the random seeds of keras and train_test_split, therefore, you will reproduce the scenario in which the trained model constantly predicts zero.

In fact, as you mentioned in your post, similar scenarios are not rare (and try a shallower model does not help), I think the problem is Dense is simply not capable of dealing with time series, you need LSTM instead. Try the code below, in which I replaced Dense with LSTM+Dropout, besides, I changed the activation function of the output layer to tanh.

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler
from tensorflow.keras.models import Model,Sequential
from tensorflow.keras.layers import Dense,LSTM,Dropout

from matplotlib import pyplot as plt
import numpy as np
import tensorflow as tf
from tensorflow import keras

# main
def series_to_supervised(data,n_in,n_out):

df = pd.DataFrame(data)
cols = list()
for i in range(n_in,0,-1): cols.append(df.shift(i))
for i in range(0, n_out): cols.append(df.shift(-i))
agg = pd.concat(cols,axis=1)
agg.dropna(inplace=True)

return agg.values

n_in,n_out = 14,1
scaler = MinMaxScaler(feature_range=(0, 1))
scaler = scaler.fit(data)
scaled_data=scaler.transform(data)
DATA = series_to_supervised(scaled_data[:-10], n_in, n_out)
X, Y = DATA[:, :-n_out], DATA[:, n_in:]
X_train,X_test,y_train,y_test = train_test_split(X,Y,test_size=0.1,random_state=49)
X_train = X_train[:,None,:]
X_test = X_test[:,None,:]

wrong_ind = 0
for ind in range(100):
print('working on %i' % ind)
keras.backend.clear_session()
model = Sequential()

# fit
n_epoch = 5 if ind < 99 else 200
history = model.fit(X_train,y_train,validation_data=(X_test,y_test),epochs=n_epoch,verbose=0)
val = history.history['val_loss']
if np.abs(val[0] - val[-1]) < 1e-4:
print(ind,val)
wrong_ind += 1

print(wrong_ind)

fig,ax = plt.subplots(nrows=1,ncols=2,figsize=(12,8))
ax[0].plot(history.history['val_loss'],'r')
ax[0].plot(history.history['loss'],'b')
ax[1].plot(model.predict(X[:,None,:]),'r')
ax[1].plot(Y,'b')
plt.show()


the output is

• I have shallow the model like you but the problem still persist. Can you post on some github, your whole code and data.pny? In order I reproduce it. – Paul Nov 11 '20 at 10:56
• @Paul Sure, I’m currently not with my Mac, I will include data.npy later. Actually, the data.npy is just your data array as I mentioned in my post, but I will include it anyway. – meTchaikovsky Nov 11 '20 at 11:01
• @Paul I have uploaded data.npy, check out my updated post. – meTchaikovsky Nov 11 '20 at 11:17
• Indeed, I have just try your code and 35/100 the model doesn't learn anything. – Paul Nov 11 '20 at 20:06
• @Paul Check out my updated post, given the nature of your data, use LSTM resolves the issue :) – meTchaikovsky Nov 12 '20 at 4:54