How to deal with severe overfitting in a UNet Encoder/Decoder CNN in a task very similar to image translation?

Question

I am trying to fit a UNet CNN to a task very similar to image to image translation. The input to the network is a binary matrix of size (64,256) and the output is of size (64,32). The columns represent a status of a communication channel where each entry in the column is the status of a subchannel. 1 means that the subchannel is occupied and 0 means that the subchannel is vacant. The horizontal axis represents the flow of time. So, the first column is status of the channel at time slot 1 and the second column is the status at time slot and so forth. The task is to predict the status of the channel in the next 32 time slots given the the previous 256 time slots which I treated as image to image translation. The accuracy on the training data is around 90% while the accuracy on the test is around 50%. By accuracy here, I mean the average percentage of correct entries in each image. Also, while training the validation loss increases while the loss decreases which is a clear sign of overfitting. I have tried most of the regularization techniques and also tried reducing the capacity of the model but this only reduces the training error while not improving the generalization error. Any advice or ideas? I included in the next part the learning curve for training on 1000 samples, the implementation of the network and samples from the training and test sets.

Learning curves of training on 1000 samples

3 Samples from the training set

3 Samples From the test set

Here is the implementation of the network:

def define_encoder_block(layer_in, n_filters, batchnorm=True):
    # weight initialization
    init = RandomNormal(stddev=0.02)
    # add downsampling layer
    g = Conv2D(n_filters, (4,4), strides=(2,2), padding='same',
               kernel_initializer=init)(layer_in)
    # conditionally add batch normalization
    if batchnorm:
        g = BatchNormalization()(g, training=True)
    # leaky relu activation
    g = LeakyReLU(alpha=0.2)(g)
    return g
 
# define a decoder block
def decoder_block(layer_in, skip_in, n_filters, filter_strides, dropout=True, skip=True):
  # weight initialization
  init = RandomNormal(stddev=0.02)
    # add upsampling layer
  g = Conv2DTranspose(n_filters, (4,4), strides=filter_strides, padding='same', 
                         kernel_initializer=init)(layer_in)
    # add batch normalization
  g = BatchNormalization()(g, training=True)
    # conditionally add dropout
  if dropout:
    g = Dropout(0.5)(g, training=True)
  if skip:
    g = Concatenate()([g, skip_in])
    # relu activation
  g = Activation('relu')(g)
  return g
 
# define the standalone generator model
def define_generator(image_shape=(64,256,1)):
    # weight initialization
    init = RandomNormal(stddev=0.02)
    # image input
    in_image = Input(shape=image_shape)
    e1 = define_encoder_block(in_image, 64, batchnorm=False)
    e2 = define_encoder_block(e1, 128)
    e3 = define_encoder_block(e2, 256)
    e4 = define_encoder_block(e3, 512)
    e5 = define_encoder_block(e4, 512)
    e6 = define_encoder_block(e5, 512)
    e7 = define_encoder_block(e6, 512)
    # bottleneck, no batch norm and relu
    b = Conv2D(512, (4,4), strides=(2,2), padding='same', kernel_initializer=init)(e7)
    b = Activation('relu')(b)
    # decoder model
    d1 = decoder_block(b, e7, 512, (1,2))
    d2 = decoder_block(d1, e6, 512, (1,2))
    d3 = decoder_block(d2, e5, 512, (2,2))
    d4 = decoder_block(d3, e4, 512, (2,2), dropout=False)
    d5 = decoder_block(d4, e3, 256, (2,2), dropout=False)
    d6 = decoder_block(d5, e2, 128, (2,1), dropout=False, skip= False)
    d7 = decoder_block(d6, e1, 64, (2,1), dropout=False,  skip= False)
    # output
    g = Conv2DTranspose(1, (4,4), strides=(2,1), padding='same', kernel_initializer=init)(d7)
    out_image = Activation('sigmoid')(g)
    # define model
    model = Model(in_image, out_image)
    return model

Answer 1

I'm not so sure that what you're seeing above is actually an overfitting problem what IM seeing is that the validation curve isn't actually going down at all, hard to say that this is overfitting, typically in the case of overfitting we see the validation loss will come down, but at a certain point it'll start to go back up

All of this is indicating to me that either the validation set doesn't look like the training set, the model doesn't have the capacity to actually learn the patterns, but is capable of memorizing it (this is actually a subtle bias problem)

One problem I also see w/your implementation is that you first BatchNorm, than apply dropout, dropout is applied post-activations, so move that around. This could be causing issues

For starters, what happens if the regularization is reduced entirely? Lets say you remove the dropout and the batchnorm terms?