2
$\begingroup$

The Issue

I've been working on a regression CNN implementation to predict time series data and have run into an issue where my validation loss and training loss diverge immediately during training, as seen below:

enter image description here

enter image description here

Usually when validation loss increases during training overfitting is the culprit, but in this case the validation loss doesn't seem to decrease initially at all which is weird. I have tried treating this with the normal fixes for overfitting, i.e increasing dropout and increasing the amount of data, but to no avail.

For reference, the model above was trained with approx. 60,000 samples as shown in the confusion matrix (more on this in a second), but I also trained a model with over 250,000 samples and got the same result: enter image description here

enter image description here

For this reason it seems unlikely to me that overfitting is the sole problem, as I would think that increasing the data by so much would have some real effect.

The Details

Overview

This project is an attempt to perform time series prediction with a CNN by creating images from feature data. The data has 28 separate features, and a 28x28 image is created by using a 28 day window to capture those features (making each row of pixels 28 days of that feature time series). The features themselves are related, but not necessary in the same magnitude (ex, some are around 0-1, some in the 100's, some include negatives)

Each image has a label which is a value between -1 and 1, and the sign of the label is of specific importance, so this problem could also be framed as a classification problem by taking the sign as a class (this is why the confusion matrices were created as a metric). For this reason I am also noting that the dataset is balanced, with almost exactly 50% positive/negative. I have also tested this project using classification, but the problem still persists.

Training Images

Below are some examples of the images I'm generating as well as a sample of the values within (different) images before and after normalization.

enter image description here enter image description here enter image description here

Pre-Processing

Each feature time series is normalized between 0,1 within the scope of each image with the following code. My first assumption here is that since my feature data has trends, it would be better to normalize within the image rather than over the entire length of the dataset (making the later images have higher values). My second assumption is that it's impossible to normalize the entire image array at once (ex. dividing by 255 for MNIST data) since each row of pixels is of a different scale.

def normalizeImages(dataset):
imageList = dataset['trainingImages'].values
for i in range(len(imageList)):
    image = imageList[i]
    for j in range(len(image)):
        row = image[j].reshape(len(image[j]),1)
        minmaxScaler = MinMaxScaler(feature_range=(0, 1))  # 0,1 seems to be standard for image data
        minmaxScaler.fit(row)
        row = minmaxScaler.transform(row)
        image[j] = row.squeeze()
    imageList[i] = image
dataset['trainingImages'] = imageList
return dataset

It's worth noting that this is applied to all images in exactly the same way before they are split into validation/testing sets and then shuffled.

In addition to normalizing the images, the features themselves are arranged within the images using clustering to group similar series together on the Y axis to increase locality.

Model

The model being used is based off of some examples for MNIST classification:

 model = Sequential([
    Conv2D(32, (3, 3), input_shape=(inputDim, inputDim, 1), activation='relu'),
    Conv2D(64, (3, 3), activation='relu'),
    MaxPooling2D(pool_size=(4, 4)),
    Flatten(),
    Dense(128, activation='relu'),
    Dropout(0.25),
    Dense(64, activation='relu'),
    Dropout(0.5),
    Dense(1,activation='linear')
])
model.compile(loss='mse', optimizer=keras.optimizers.Adadelta(), metrics=['mse',tf.keras.metrics.RootMeanSquaredError()])

I've trained this model with epochs from 100-3000, as well as batch sizes from 32-3500 with no different results.

What Else I've Tried

Below are a few of the other things I've tried, but there are more so feel free to ask for any other details.

  • Normalizing images on the entire feature time series' rather than just within images
  • Using first differences on the time series to remove trend and then normalizing on whole dataset/within image
  • Training on MNIST data, the model (modified for classification) learns it to 98% accuracy.
  • Changing Optimizers to ADAM
  • Changing learning rate for ADADELTA: going from 0.001 to 0.0001 prevents learning entirely
  • Outputting loss on a per-batch basis rather than per epoch to see if I was just overfitting in the first couple epochs, but it didnt look like it.

My Theories

  • My images aren't being normalized well, so they aren't conveying data to be learned , even though the model is able to learn the training sets with 250,000 samples?
  • The testing set images are somehow different than the training images, though theyre normalized in the exact same way.
  • I'm still overfitting somehow, need more data/dropout/other?

Concluding Thoughts

I am by no means an expert in this field, so it is very possible that I've made some kind of assumption/error about normalization and image processing that prevents my model from actually learning from my images.

I've tried to include as much information as possible that I thought would be relevant to my issue, but I'm happy to provide anything else upon request.

Thanks in advance for any suggestions and knowledge you can give to help me troubleshoot this issue!

Edits

In response to etiennedm

I changed my dense 64 layer to the following:

Dense(64, activation='tanh'),

but the issue still persists unfortunately: enter image description here

I had tried running without any dropout before but that didn't make a difference as the model seems to be learning the training data fine, I assume removing dropout would only make it overfit faster? enter image description here

This is the result when the 28 feature time series are normalized on their full data rather than within the 28 day images only. I use the following code to do this:

minmaxScaler = MinMaxScaler(feature_range=(0,1)) 
    minmaxScaler.fit(trainingSample) 
    featureData = minmaxScaler.transform(featureData)

It may be worth noting that the scaler is fit on only the training data and then applied to the whole set to eliminate data leakage between train/test sets. Still the result seems to be about the same:

enter image description here

$\endgroup$
0
$\begingroup$

Using a relu function at the n-1 layer could be too constraining if you want your network to produce both positive and negative values. I am not sure about your image preprocessing, but I would first give a try to change (at least) the last activation function relu to leaky relu or tanh (an activation function that produce both positive and negative values).

Note: you could also remove the dropout until your model actually learns. Once it is learning, add dropout for regularization purpose.

EDIT

You could try to normalize your 28 features as features are normally normalized. That means in your training set, take all the values for each feature and create a scaler that you apply then on your testing images.

$\endgroup$
4
  • $\begingroup$ Thanks for the suggestions! I edited the question to add the results of those tests but unfortunately it doesn't seem to have changed anything. $\endgroup$
    – alanbevel
    Sep 22 '20 at 14:43
  • $\begingroup$ In response to the edit I did a test by normalizing along each feature set before creating the images(which can be found above). This should give the same result as fitting a scaler for each time series and then applying it within that row of the images. One alternative way I read your suggestion was to make one scaler and fit it on every time series before applying it to the whole image, but I don't think that will work due to the huge difference in scale between the features (i.e some in the millions, some just floats, some negatives) $\endgroup$
    – alanbevel
    Sep 22 '20 at 17:24
  • $\begingroup$ I agree this is unexpected. There are a few possibilities. The features could not carry any information. The NN structure is not good to extract the right information. Or we are missing something in your preprocessing. Do you think you could share your data/code so we can dig further ? $\endgroup$
    – etiennedm
    Sep 22 '20 at 20:18
  • $\begingroup$ Sure, though I'm not sure of the best way to do that here, I'll contact you at the email in your profile and send some more information that way if that works for you. $\endgroup$
    – alanbevel
    Sep 22 '20 at 20:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.