Restricting the output of a model didn't improve the loss value of the model evaluation

Question

There is a deep model for prediction.

The outputs are some numbers between 0 and 80. (In the dataset the outputs are 0-80)

The model Loss value is 70 and I would like to reduce it.

I printed the outputs after evaluating the model by test values and some of the predicted values are more than 80 or less than 0.

I decided to set up the final layer to predict just in 0-80 in the training step, therefore I set a lambda layer after final Dense layer to clip output values.

The codes:

def relu_advanced(x):
return K.relu(x, max_value=80)

def createModel4():
model = models.Sequential()
model.add(Conv2D(256,(3, 3),
                 activation='relu',
                 input_shape=(320,20,1), padding='same'))
model.add(MaxPooling2D((2, 2)))

model.add(Conv2D(128, (3, 3), activation='relu', padding='same'))
model.add(MaxPooling2D((2, 2)))

model.add(Conv2D(64, (3, 3), activation='relu', padding='same'))
model.add(MaxPooling2D((2,2)))

model.add(Flatten())
#model.add(Dense(5*320, activation= 'relu'))
model.add(Dense(5*320))
model.add(Lambda(relu_advanced))



model.summary()
return model

I tested the model with and without the relu_advanced and unfortunately, the Loss value is increased with advanced_relu!

While there is no value much than 80 or less than zero, I don't know what may happen that the Loss is increased?

Thank you

Answer 1

I suggest you normalise your labels so that it is scaled between 0 and 1, rather than 0 and 80. Once you have a trained model then multiply your output at the end. The network should find it easier to learn values between 0 and 1 (Andrew Ng's coursera course has a good lecture on this).

Go back to using the standard ReLU: The problem with yours is that it cannot learn if it tries to output a value greater than 80, as there is no gradient with which to update the parameters.

I would also consider putting in a much smaller Dense layer at the end (eg 10 nodes).