Why does my CNN validation loss increase immediately, even with lots of data?

Question

The Issue

I've been working on a regression CNN implementation to predict time series data and have run into an issue where my validation loss and training loss diverge immediately during training, as seen below:

Usually when validation loss increases during training overfitting is the culprit, but in this case the validation loss doesn't seem to decrease initially at all which is weird. I have tried treating this with the normal fixes for overfitting, i.e increasing dropout and increasing the amount of data, but to no avail.

For reference, the model above was trained with approx. 60,000 samples as shown in the confusion matrix (more on this in a second), but I also trained a model with over 250,000 samples and got the same result:

For this reason it seems unlikely to me that overfitting is the sole problem, as I would think that increasing the data by so much would have some real effect.

The Details

Overview

This project is an attempt to perform time series prediction with a CNN by creating images from feature data. The data has 28 separate features, and a 28x28 image is created by using a 28 day window to capture those features (making each row of pixels 28 days of that feature time series). The features themselves are related, but not necessary in the same magnitude (ex, some are around 0-1, some in the 100's, some include negatives)

Each image has a label which is a value between -1 and 1, and the sign of the label is of specific importance, so this problem could also be framed as a classification problem by taking the sign as a class (this is why the confusion matrices were created as a metric). For this reason I am also noting that the dataset is balanced, with almost exactly 50% positive/negative. I have also tested this project using classification, but the problem still persists.

Training Images

Below are some examples of the images I'm generating as well as a sample of the values within (different) images before and after normalization.

Pre-Processing

Each feature time series is normalized between 0,1 within the scope of each image with the following code. My first assumption here is that since my feature data has trends, it would be better to normalize within the image rather than over the entire length of the dataset (making the later images have higher values). My second assumption is that it's impossible to normalize the entire image array at once (ex. dividing by 255 for MNIST data) since each row of pixels is of a different scale.

def normalizeImages(dataset):
imageList = dataset['trainingImages'].values
for i in range(len(imageList)):
    image = imageList[i]
    for j in range(len(image)):
        row = image[j].reshape(len(image[j]),1)
        minmaxScaler = MinMaxScaler(feature_range=(0, 1))  # 0,1 seems to be standard for image data
        minmaxScaler.fit(row)
        row = minmaxScaler.transform(row)
        image[j] = row.squeeze()
    imageList[i] = image
dataset['trainingImages'] = imageList
return dataset

It's worth noting that this is applied to all images in exactly the same way before they are split into validation/testing sets and then shuffled.

In addition to normalizing the images, the features themselves are arranged within the images using clustering to group similar series together on the Y axis to increase locality.

Model

The model being used is based off of some examples for MNIST classification:

 model = Sequential([
    Conv2D(32, (3, 3), input_shape=(inputDim, inputDim, 1), activation='relu'),
    Conv2D(64, (3, 3), activation='relu'),
    MaxPooling2D(pool_size=(4, 4)),
    Flatten(),
    Dense(128, activation='relu'),
    Dropout(0.25),
    Dense(64, activation='relu'),
    Dropout(0.5),
    Dense(1,activation='linear')
])
model.compile(loss='mse', optimizer=keras.optimizers.Adadelta(), metrics=['mse',tf.keras.metrics.RootMeanSquaredError()])

I've trained this model with epochs from 100-3000, as well as batch sizes from 32-3500 with no different results.

What Else I've Tried

Below are a few of the other things I've tried, but there are more so feel free to ask for any other details.

• Normalizing images on the entire feature time series' rather than just within images
• Using first differences on the time series to remove trend and then normalizing on whole dataset/within image
• Training on MNIST data, the model (modified for classification) learns it to 98% accuracy.
• Changing Optimizers to ADAM
• Changing learning rate for ADADELTA: going from 0.001 to 0.0001 prevents learning entirely
• Outputting loss on a per-batch basis rather than per epoch to see if I was just overfitting in the first couple epochs, but it didnt look like it.

My Theories

• My images aren't being normalized well, so they aren't conveying data to be learned , even though the model is able to learn the training sets with 250,000 samples?
• The testing set images are somehow different than the training images, though theyre normalized in the exact same way.
• I'm still overfitting somehow, need more data/dropout/other?

Concluding Thoughts

I am by no means an expert in this field, so it is very possible that I've made some kind of assumption/error about normalization and image processing that prevents my model from actually learning from my images.

I've tried to include as much information as possible that I thought would be relevant to my issue, but I'm happy to provide anything else upon request.

Thanks in advance for any suggestions and knowledge you can give to help me troubleshoot this issue!

Edits

In response to etiennedm

I changed my dense 64 layer to the following:

Dense(64, activation='tanh'),

but the issue still persists unfortunately:

I had tried running without any dropout before but that didn't make a difference as the model seems to be learning the training data fine, I assume removing dropout would only make it overfit faster?

This is the result when the 28 feature time series are normalized on their full data rather than within the 28 day images only. I use the following code to do this:

minmaxScaler = MinMaxScaler(feature_range=(0,1)) 
    minmaxScaler.fit(trainingSample) 
    featureData = minmaxScaler.transform(featureData)

It may be worth noting that the scaler is fit on only the training data and then applied to the whole set to eliminate data leakage between train/test sets. Still the result seems to be about the same:

Answer 1

Using a relu function at the n-1 layer could be too constraining if you want your network to produce both positive and negative values. I am not sure about your image preprocessing, but I would first give a try to change (at least) the last activation function relu to leaky relu or tanh (an activation function that produce both positive and negative values).

Note: you could also remove the dropout until your model actually learns. Once it is learning, add dropout for regularization purpose.

EDIT

You could try to normalize your 28 features as features are normally normalized. That means in your training set, take all the values for each feature and create a scaler that you apply then on your testing images.