3
$\begingroup$

I have been trying to do some sort of image enhancement on grayscale images. I have used both pixel wise loss and perceptual loss (perceptual loss uses classifier between 2 classes trained on the same dataset).

So the input to my network is an bad image and the output is the enhanced image where edges in an image are clearer.

I have written the code in keras

inputs = Input(shape=image_shape)

x = Conv2D(filters=ngf, kernel_size=(7, 7), padding='same')(inputs)
x = BatchNormalization()(x)
x = LeakyReLU(0.2)(x)

n_downsampling = 2
for i in range(n_downsampling):
    mult = 2**i
    x = Conv2D(filters=ngf*mult*2, kernel_size=(3, 3), strides=2, padding='same')(x)
    x = BatchNormalization()(x)
    x = LeakyReLU(0.2)(x)

mult = 2**n_downsampling
for i in range(n_blocks_gen):
    x = res_block(x, ngf*mult, use_dropout=False)

for i in range(n_downsampling):
    mult = 2**(n_downsampling - i)
    x = Conv2DTranspose(filters=int(ngf * mult / 2), kernel_size=(3, 3), strides=2, padding='same')(x)
    x = BatchNormalization()(x)
    x = LeakyReLU(0.2)(x)

#x = ReflectionPadding2D((3, 3))(x)
x = Conv2D(filters=output_nc, kernel_size=(5, 5), padding='same')(x)
x = Conv2D(filters=output_nc, kernel_size=(3, 3))(x)
x = Activation('sigmoid')(x)

outputs = Add()([x, inputs])
#outputs = Lambda(lambda z: K.clip(z, -1, 1))(x)
outputs = Lambda(lambda z: z/2)(outputs)

model = Model(inputs=inputs, outputs=outputs, name='Generator')

Function for res_block

 def res_block(input, filters, kernel_size=(3, 3), strides=(1, 1), 
 use_dropout=False):
"""
Instanciate a Keras Resnet Block using sequential API.

:param input: Input tensor
:param filters: Number of filters to use
:param kernel_size: Shape of the kernel for the convolution
:param strides: Shape of the strides for the convolution
:param use_dropout: Boolean value to determine the use of dropout
:return: Keras Model
"""
x = Conv2D(filters=filters,
           kernel_size=kernel_size,
           strides=strides, 
           padding='same')(input)
x = BatchNormalization()(x)
x = Activation('relu')(x)

if use_dropout:
    x = Dropout(0.5)(x)

x = Conv2D(filters=filters,
           kernel_size=kernel_size,
           strides=strides, 
           padding='same')(x)
x = BatchNormalization()(x)

merged = Add()([input, x])
return merged

Any idea why this wouldn't overfit ? I have been looking for other loss functions but I couldn't find much. I personally tried to make pixel loss penalize the model on pixels where it should have completed a incomplete line in an image by using sobel operator on the enhanced image to get a mask which is multiplied by the difference between the input & target image hoping that the network would focus on completing incomplete edges or so but nothing changed.

Improve this question
$\endgroup$
8
  • $\begingroup$ Why do you want to overfit? Did you try training longer? $\endgroup$
    – Valentin Calomme
    Jul 31, 2018 at 13:30
  • $\begingroup$ Without seeing what the res_block function looks like it’s hard to say. Also most people apply batchnorm right after ReLU, not right before. $\endgroup$
    – kbrose
    Jul 31, 2018 at 13:54
  • $\begingroup$ Also can we see how you compile the model? $\endgroup$
    – kbrose
    Jul 31, 2018 at 13:54
  • $\begingroup$ @kbrose the compilation is just a regular model.compile('rmsprop',loss='mse'). $\endgroup$
    – Karim Mohamed Hasebou
    Jul 31, 2018 at 14:29
  • 1
    $\begingroup$ @ValentinCalomme usually I over-fit on a sample of the data before I train on the whole data to make sure everything is okay. $\endgroup$
    – Karim Mohamed Hasebou
    Jul 31, 2018 at 14:30

1 Answer 1

Reset to default
2
$\begingroup$

You use a sigmoid activation in the last layer, which restricts your possible outputs to be between 0 and 1. This is typically used for binary classification, but it seems like you are not doing classification at all. If your image has values constrained to the range 0-1 then this is likely fine. Otherwise, this could be the source of your problem.

I suggest removing the sigmoid activation in the final convolution and leaving it with no activation.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I perform pre-processing to ensure that the range of the image is between 0 and 1. Is it possible that because the image am training on contain a lot of lines, that mse is unable to cope with so many changes that it always ends up converging to gray empty image ? $\endgroup$
    – Karim Mohamed Hasebou
    Aug 2, 2018 at 18:30
  • $\begingroup$ Ok, if you're scaling then sigmoid may be fine. Near 0 and 1 the derivative of sigmoid is quite small, so you could be running into problems there. Do you have a lot of values near 0 and 1? $\endgroup$
    – kbrose
    Aug 2, 2018 at 20:45
  • $\begingroup$ I would also advise letting the network train a little longer. Converging to gray sounds like it might be converging to the mean value, sometimes networks are able to escape predicting the mean if you let them churn a little longer. $\endgroup$
    – kbrose
    Aug 2, 2018 at 20:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.