3
$\begingroup$

I'm building a lstm model for regression on timeseries. To verify my implementation of the model and understand keras, I'm using a toyproblem to make sure I understand what's going on. Problem is I do not understand what's going on here.

As I am fitting the model, training loss is constantly larger than validation loss, even for a balanced train/validation set (5000 samples each):

Model Loss

In my understanding the two curves should be exactly the other way around such that training loss would be an upper bound for validation loss.

Predictions are more or less ok here. However I'd still like to understand what's going on, as I see similar behavior of the loss in my real problem but there the predictions are rubbish. So I suspect, there's something going on with the model that I don't understand.

Here's the code for my toy problem:

import numpy as np
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout
from keras.layers import LSTM
import matplotlib.pyplot as plt
from sklearn.preprocessing import MinMaxScaler

#create testdata
nEpochs = 12
nTimestepsPerSeq = 5
nFeatures = 5

def generate_examples(nSamples, nTimestepsPerSeq, nFeatures):
    X = np.random.random((nSamples, nTimestepsPerSeq, nFeatures))

    #make feature 1 categorical: [0,1,2]
    X[:,:,0] = np.random.randint(0,3, X[:,:,0].shape)

    #make feature 2 categorical: [-1, 0,1]
    X[:,:,1] = np.random.randint(-1,2, X[:,:,1].shape)

    #shift feature 3 by a constant
    X[:,:,2] = X[:,:,2] + 2

    #calc output
    Y = np.zeros((1, nSamples))

    #combine features and introduce non-linearity
    Y = X[:,-1,0]*np.mean(X[:,-1,3]) + X[:,-1,2]*np.mean(X[:,-1,4]) + \
        (X[:,-1,0]*X[:,-1,1]*np.mean(X[:,-1,2]))**2

    #add uniform noise
    Y = Y*np.random.uniform(0.95,1.05,size=Y.shape)

    #reshape for scaler instance:
    # ValueError: Expected 2D array, got 1D array instead:
    # array=[  1.27764489  27.56604355   1.39317709 ...,   1.57210734   8.18834281
    # 1.66174279].
    # Reshape your data either using array.reshape(-1, 1) if your data has a single fe
    # ature or array.reshape(1, -1) if it contains a single sample.
    Y = Y.reshape((-1,1))

    return X,Y

Xtrain,Ytrain = generate_examples(5000, nTimestepsPerSeq, nFeatures)
Xval,Yval = generate_examples(5000, nTimestepsPerSeq, nFeatures)
Xtest,Ytest = generate_examples(20, nTimestepsPerSeq, nFeatures)

#scale input data
for i in range(0,nFeatures):
    #scaler = StandardScaler()
    scaler = MinMaxScaler()
    scaler = scaler.fit(Xtrain[:,:,i])
    Xtrain[:,:,i] = scaler.transform(Xtrain[:,:,i])
    Xval[:,:,i] = scaler.transform(Xval[:,:,i])
    Xtest[:,:,i] = scaler.transform(Xtest[:,:,i])

targetScaler = MinMaxScaler()
targetScaler = targetScaler.fit(Ytrain)

#transform target
Ytrain = targetScaler.transform(Ytrain)    
Yval = targetScaler.transform(Yval)    
Ytest = targetScaler.transform(Ytest) 

# defining the LSTM model
model = Sequential()
model.add(LSTM(200, input_shape=(Xtrain.shape[1], Xtrain.shape[2]), return_sequences=True))
model.add(LSTM(200))
model.add(Dense(1))

model.compile(loss='mse', optimizer='adam', metrics=['acc'])

# fitting the model
history = model.fit(Xtrain, Ytrain, epochs=nEpochs, batch_size=50, validation_data=(Xval, Yval), shuffle=True, verbose=2)

#test model
yhat = model.predict(Xtest)
print("pediction vs truth:")
for i in range(0,10):
    print(yhat[i], Ytest[i])

# summarize history for loss
plt.subplot(1,1,1)
plt.plot(history.history['loss'], '.-')
plt.plot(history.history['val_loss'], '.-')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper right')
plt.show()

Edit: I added some output of an experiment:

Epoch 1/12
 - 6s - loss: 0.1056 - acc: 2.0000e-04 - val_loss: 0.0680 - val_acc: 0.0000e+00
Epoch 2/12
...
Epoch 11/12
 - 4s - loss: 0.0033 - acc: 4.0000e-04 - val_loss: 0.0020 - val_acc: 0.0000e+00
Epoch 12/12
 - 4s - loss: 0.0016 - acc: 4.0000e-04 - val_loss: 0.0016 - val_acc: 0.0000e+00
pediction vs truth:
[ 0.25022525] [ 0.25465108]
[ 0.98761547] [ 0.91661543]
[ 1.06177747] [ 0.95979166]
[ 0.0835482] [ 0.03742919]
[ 0.02432941] [ 0.01149685]
[ 0.00915699] [ 0.00887351]
[ 0.2765356] [ 0.27340488]
[ 0.02941256] [ 0.01685951]
[-0.01059875] [ 0.00157809]
[ 0.04762106] [ 0.01983566]
$\endgroup$
  • 1
    $\begingroup$ Is this line correct? targetScaler = scaler.fit(Ytrain). Should it not be targetScaler.fit()? $\endgroup$ – n1k31t4 Jun 11 '18 at 11:39
  • $\begingroup$ If I run your code (unchanged - on a GPU), then the model doesn't seem to train. I get NaN values for train/val loss and therefore 0.0% accuracy. Did you need to set anything else? $\endgroup$ – n1k31t4 Jun 11 '18 at 12:05
  • $\begingroup$ thank you n1k31t4 for your replies, you're right about the scaler/targetScaler issue, however it doesn't significantly change the outcome of the experiment. I just copied the code above (fixed the scaler bug) and reran it on CPU. I am getting different values for the loss function per epoch. However I don't get any sensible values for accuracy. I just attributed that to a poor choice for the accuracy-metric and haven't given it much thought. I edited my original post to accomodate your input and some information about my loss/acc values $\endgroup$ – Mischa Obrecht Jun 11 '18 at 16:19
2
$\begingroup$

Training scores can be expected to be better than those of the validation when the machine you train can "adapt" to the specifics of the training examples while not successfully generalizing; the greater the adaption to the specifics of the training examples and the worse generalization, the bigger the gap between training and validation scores (in favor of the training scores).

In cases in which training as well as validation examples are generated de novo, the network is not presented with the same examples over and over. It thus cannot overfit to accommodate them while losing the ability to respond correctly to the validation examples - which, after all, are generated by the same process as the training examples. For an example of such an approach you can have a look at my experiment.

As you commented, this in not the case here, you generate the data only once. So this does not explain why you do not see overfit. Other explanations might be that this is because your network does not have enough trainable parameters to overfit, coupled with a relatively large number of training examples (and of course, generating the training and the validation examples with the same process). It might also be possible that you will see overfit if you invest more epochs into the training. Might be an interesting experiment.

Okay, so this explains why the validation score is not worse. But why is it better?

Be advised that validation, as it is calculated at the end of each epoch, uses the "best" machine trained in that epoch (that is, the last one, but if constant improvement is the case then the last weights should yield the best results - at least for training loss, if not for validation), while the train loss is calculated as an average of the performance per each epoch.

Thus, if the machine is constantly improving and does not overfit, the gap between the network's average performance in an epoch and its performance at the end of an epoch is translated into the gap between training and validation scores - in favor of the validation scores.

Give or take minor variations that result from the random process of sample generation (even if data is generated only once, but especially if it is generated anew for each epoch).

$\endgroup$
  • $\begingroup$ Thanks a bunch for your insight! The second part makes sense to me, however in the first part you say, I am creating examples de novo, but I am only generating the data once. I think I might have misunderstood something here, what do you mean exactly by "the network is not presented with the same examples over and over"? $\endgroup$ – Mischa Obrecht Jun 11 '18 at 19:25
  • $\begingroup$ Ok, rereading your code I can obviously see that you are correct; I will edit my answer. Seeing as you do not generate the examples anew every time, it is reasonable to assume that you would reach overfit, given enough epochs, if it has enough trainable parameters. $\endgroup$ – Lafayette Jun 12 '18 at 8:28
  • $\begingroup$ I just tried increasing the number of training epochs to 50 (instead of 12) and the number of neurons per layer to 500 (instead of 100) and still couldn't get the model to overfit. However training as well as validation loss pretty much converge to zero, so I guess we can conclude that the problem is to easy because training and validation data are generated in exactly the same way. $\endgroup$ – Mischa Obrecht Jun 12 '18 at 16:57
  • $\begingroup$ Thank you for informing me regarding your experiment. I agree with your analysis. $\endgroup$ – Lafayette Jun 13 '18 at 8:14
  • 1
    $\begingroup$ @Lafayette, alas, the link you posted to your experiment is broken $\endgroup$ – Riley Jun 12 '19 at 8:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.