2
$\begingroup$

There is a deep model for prediction.

The outputs are some numbers between 0 and 80. (In the dataset the outputs are 0-80)

The model Loss value is 70 and I would like to reduce it.

I printed the outputs after evaluating the model by test values and some of the predicted values are more than 80 or less than 0.

I decided to set up the final layer to predict just in 0-80 in the training step, therefore I set a lambda layer after final Dense layer to clip output values.

The codes:

def relu_advanced(x):
return K.relu(x, max_value=80)

def createModel4():
model = models.Sequential()
model.add(Conv2D(256,(3, 3),
                 activation='relu',
                 input_shape=(320,20,1), padding='same'))
model.add(MaxPooling2D((2, 2)))

model.add(Conv2D(128, (3, 3), activation='relu', padding='same'))
model.add(MaxPooling2D((2, 2)))

model.add(Conv2D(64, (3, 3), activation='relu', padding='same'))
model.add(MaxPooling2D((2,2)))

model.add(Flatten())
#model.add(Dense(5*320, activation= 'relu'))
model.add(Dense(5*320))
model.add(Lambda(relu_advanced))



model.summary()
return model

I tested the model with and without the relu_advanced and unfortunately, the Loss value is increased with advanced_relu!

While there is no value much than 80 or less than zero, I don't know what may happen that the Loss is increased?

Thank you

$\endgroup$
1
$\begingroup$

I suggest you normalise your labels so that it is scaled between 0 and 1, rather than 0 and 80. Once you have a trained model then multiply your output at the end. The network should find it easier to learn values between 0 and 1 (Andrew Ng's coursera course has a good lecture on this).

Go back to using the standard ReLU: The problem with yours is that it cannot learn if it tries to output a value greater than 80, as there is no gradient with which to update the parameters.

I would also consider putting in a much smaller Dense layer at the end (eg 10 nodes).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.