I have been trying to do some sort of image enhancement on grayscale images. I have used both pixel wise loss and perceptual loss (perceptual loss uses classifier between 2 classes trained on the same dataset).

So the input to my network is an bad image and the output is the enhanced image where edges in an image are clearer.

I have written the code in keras

inputs = Input(shape=image_shape)

x = Conv2D(filters=ngf, kernel_size=(7, 7), padding='same')(inputs)
x = BatchNormalization()(x)
x = LeakyReLU(0.2)(x)

n_downsampling = 2
for i in range(n_downsampling):
    mult = 2**i
    x = Conv2D(filters=ngf*mult*2, kernel_size=(3, 3), strides=2, padding='same')(x)
    x = BatchNormalization()(x)
    x = LeakyReLU(0.2)(x)

mult = 2**n_downsampling
for i in range(n_blocks_gen):
    x = res_block(x, ngf*mult, use_dropout=False)

for i in range(n_downsampling):
    mult = 2**(n_downsampling - i)
    x = Conv2DTranspose(filters=int(ngf * mult / 2), kernel_size=(3, 3), strides=2, padding='same')(x)
    x = BatchNormalization()(x)
    x = LeakyReLU(0.2)(x)

#x = ReflectionPadding2D((3, 3))(x)
x = Conv2D(filters=output_nc, kernel_size=(5, 5), padding='same')(x)
x = Conv2D(filters=output_nc, kernel_size=(3, 3))(x)
x = Activation('sigmoid')(x)

outputs = Add()([x, inputs])
#outputs = Lambda(lambda z: K.clip(z, -1, 1))(x)
outputs = Lambda(lambda z: z/2)(outputs)

model = Model(inputs=inputs, outputs=outputs, name='Generator')

Function for res_block

 def res_block(input, filters, kernel_size=(3, 3), strides=(1, 1), 
Instanciate a Keras Resnet Block using sequential API.

:param input: Input tensor
:param filters: Number of filters to use
:param kernel_size: Shape of the kernel for the convolution
:param strides: Shape of the strides for the convolution
:param use_dropout: Boolean value to determine the use of dropout
:return: Keras Model
x = Conv2D(filters=filters,
x = BatchNormalization()(x)
x = Activation('relu')(x)

if use_dropout:
    x = Dropout(0.5)(x)

x = Conv2D(filters=filters,
x = BatchNormalization()(x)

merged = Add()([input, x])
return merged

Any idea why this wouldn't overfit ? I have been looking for other loss functions but I couldn't find much. I personally tried to make pixel loss penalize the model on pixels where it should have completed a incomplete line in an image by using sobel operator on the enhanced image to get a mask which is multiplied by the difference between the input & target image hoping that the network would focus on completing incomplete edges or so but nothing changed.

  • $\begingroup$ Why do you want to overfit? Did you try training longer? $\endgroup$ – Valentin Calomme Jul 31 '18 at 13:30
  • $\begingroup$ Without seeing what the res_block function looks like it’s hard to say. Also most people apply batchnorm right after ReLU, not right before. $\endgroup$ – kbrose Jul 31 '18 at 13:54
  • $\begingroup$ Also can we see how you compile the model? $\endgroup$ – kbrose Jul 31 '18 at 13:54
  • $\begingroup$ @kbrose the compilation is just a regular model.compile('rmsprop',loss='mse'). $\endgroup$ – Karim Mohamed Hasebou Jul 31 '18 at 14:29
  • $\begingroup$ @ValentinCalomme usually I over-fit on a sample of the data before I train on the whole data to make sure everything is okay. $\endgroup$ – Karim Mohamed Hasebou Jul 31 '18 at 14:30

You use a sigmoid activation in the last layer, which restricts your possible outputs to be between 0 and 1. This is typically used for binary classification, but it seems like you are not doing classification at all. If your image has values constrained to the range 0-1 then this is likely fine. Otherwise, this could be the source of your problem.

I suggest removing the sigmoid activation in the final convolution and leaving it with no activation.

  • $\begingroup$ I perform pre-processing to ensure that the range of the image is between 0 and 1. Is it possible that because the image am training on contain a lot of lines, that mse is unable to cope with so many changes that it always ends up converging to gray empty image ? $\endgroup$ – Karim Mohamed Hasebou Aug 2 '18 at 18:30
  • $\begingroup$ Ok, if you're scaling then sigmoid may be fine. Near 0 and 1 the derivative of sigmoid is quite small, so you could be running into problems there. Do you have a lot of values near 0 and 1? $\endgroup$ – kbrose Aug 2 '18 at 20:45
  • $\begingroup$ I would also advise letting the network train a little longer. Converging to gray sounds like it might be converging to the mean value, sometimes networks are able to escape predicting the mean if you let them churn a little longer. $\endgroup$ – kbrose Aug 2 '18 at 20:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.