How to use deep learning to add local (e.g. repairing) transformations to images?

Question

I want to train a neural network that removes scratches from pictures. I chose a GAN architecture with a generator (G) and a discriminator (D) and two sets of images scratchy and non-scratchy, with similar motives. G in my setting uses mainly convolutional and deconvolutional layers with ReLU. As input it uses the scratchy images. The D discriminates between the output of G and the non-scratchy images.

For example the generator would perform the following transformations:

(128, 128, 3) > (128, 128, 128) > (128, 128, 3)

Where the tuples contain (width, height, channels). Input and output need to have the same format.

But in order to get output that have same global structure as the inputs (i.e. streets, houses etc) it seem I have to use filter-sizes and strides of 1 and basically pass the complete pictures through the network.

However, the features I am looking at are rather small and local. It should be sufficient to use filters of up to 20 pixels in size for the convolutions. And then then apply the network to all parts of the input picture.

What would be a good generator architecture for such a task? Would you agree with the structure of the generator or would you expect different designs to perform better on local changes?

Here is the code for the generator that I use implemented in tensorflow.

def generator(x, batch_size, reuse=False):
    with tf.variable_scope('generator') as scope:
        if (reuse):
            tf.get_variable_scope().reuse_variables()

        s = 1
        f = 1

        assert ((WIDTH + f - 1) / s) % 1 == 0

        keep_prob = 0.5

        n_ch1 = 32
        w = init_weights('g_wc1', [f, f, CHANNELS, n_ch1])
        b = init_bias('g_bc1', [n_ch1])
        h = conv2d(x, w, s, b)
        h = bn(h, 'g_bn1')
        h = tf.nn.relu(h)
        h = tf.nn.dropout(h, keep_prob)
        h1 = h

        n_ch2 = 128
        w = init_weights('g_wc2', [f, f, n_ch1, n_ch2])
        b = init_bias('g_bc2', [n_ch2])
        h = conv2d(h, w, s, b)
        h = bn(h, "g_bn2")                
        h = tf.nn.relu(h)
        h = tf.nn.dropout(h, keep_prob)
        h2 = h

        n_ch3 = 256
        w = init_weights('g_wc3', [f, f, n_ch2, n_ch3])
        b = init_bias('g_bc3', [n_ch3])
        h = conv2d(h, w, s, b)
        h = bn(h, "g_bn3")                
        h = tf.nn.relu(h)
        h = tf.nn.dropout(h, keep_prob)      

        output_shape = [batch_size, HEIGHT//s//s, WIDTH//s//s, n_ch2]
        w = init_weights('g_wdc3', [f, f, n_ch2, n_ch3])
        b = init_bias('g_bdc3', [n_ch2])
        h = deconv2d(h, w, s, b, output_shape)        
        h = bn(h, "g_bnd3")                
        h = tf.nn.relu(h)
        h = h + h2

        output_shape = [batch_size, HEIGHT//s, WIDTH//s, n_ch1]
        w = init_weights('g_wdc2', [f, f, n_ch1, n_ch2])
        b = init_bias('g_bdc2', [n_ch1])
        h = deconv2d(h, w, s, b, output_shape)        
        h = bn(h, "g_bnd2")                
        h = tf.nn.relu(h)
        h = h + h1

        output_shape = [batch_size, HEIGHT, WIDTH, CHANNELS]
        w = init_weights('g_wdc1', [f, f, CHANNELS, n_ch1])
        b = init_bias('g_bdc1', [CHANNELS])
        h = deconv2d(h, w, s, b, output_shape)

    return tf.nn.sigmoid(h+x)

When you use a stride s>1 you will get an hourglass, where the layers get smaller but deeper. The depth is independently controlled by the n_ch variables.

BTW, I am running this on a Google colab notebook. Its amazing to have such an engine for free and be able to experiment with deep learning! Fantastic!

Answer 1

There is a paper called Spatial Transformer Networks written by Max Jaderberg et al. What it does is trying to find the canonical shape of its input by reducing transformations, like translation and rotation, or even diminishing the distortion of the inputs. It introduces a module which helps convolutional network to be spatial invariant. One of the significant achievements of this module is that it tries to enhance distorted inputs. Take a look at here.

---

I edit the answer because of the request of one of our friends. First I quote something from the popular book written by Pr. Gonzalez, I hope nothing goes wrong with copyright. Then I suggest my recommendation.

Figure 2.40, the one that I've attached, shows an example of the steps in Fig. 2.39. In this case, the transform used was the Fourier transform, which we mention briefly later in this section and discuss in detail in Chapter 4. Figure 2.40(a) is an image corrupted by sinusoidal interference and Fig. 2.40(b) is the magnitude of its Fourier transform, which is the output of the first stage in Fig. 2.39. As you will learn in Chapter 4, sinusoidal interference in the spatial domain appears as bright bursts of intensity in the transform domain. In this case, the bursts are in a circular pattern that can be seen in Fig. 2.40(b). Figure 2.40(c) shows a mask image (called a filter) with white and black representing 1 and 0, respectively. For this example, the operation in the second box of Fig. 2.39 is to multiply the mask by the transform, thus eliminating the bursts responsible for the interference. Figure 2.40(d) shows the final result, obtained by computing the inverse of the modified transform. The interference is no longer visible, and important detail is quite clear. In fact, you can even see the fiducial marks (faint crosses) that are used for image alignment.

Okay! After those quotes, I refer to my suggestion. In the ST network, the authors have claimed that their differentiable module can learn different kinds of transformations other than affine transformation. The point about that is that we usually apply two kinds of transformations. One transformation is applied to the intensity of the image, which is popular by the means of filters. The other one is called image warping where we change the position of intensities and the intensity values do not change unless they locate between the discrete grids of image entries also called pixels, picture element. Spatial transformers are good for the second task but they also can be used for the first task. There are studies about using CNNs for evaluating the images in their frequency domain and not in the spatial domain. You can employ these differentiable modules in those nets.

Finally about your specific task, what I'm seeing in your pictures, your noise has a same behaviour. I guess if that is true for all cases, your task is not learning and can be solved using usual image processing techniques.

Answer 2

Have you looked into ResNet which modifies image on a pixel level, rather than holistically modifying the image content, preserving the global structure and annotations?

Also your application sounds a lot like deep image prior.