I'm building a lstm model for regression on timeseries. To verify my implementation of the model and understand keras, I'm using a toyproblem to make sure I understand what's going on. Problem is I do not understand what's going on here.
As I am fitting the model, training loss is constantly larger than validation loss, even for a balanced train/validation set (5000 samples each):
In my understanding the two curves should be exactly the other way around such that training loss would be an upper bound for validation loss.
Predictions are more or less ok here. However I'd still like to understand what's going on, as I see similar behavior of the loss in my real problem but there the predictions are rubbish. So I suspect, there's something going on with the model that I don't understand.
Here's the code for my toy problem:
import numpy as np from keras.models import Sequential from keras.layers import Dense from keras.layers import Dropout from keras.layers import LSTM import matplotlib.pyplot as plt from sklearn.preprocessing import MinMaxScaler #create testdata nEpochs = 12 nTimestepsPerSeq = 5 nFeatures = 5 def generate_examples(nSamples, nTimestepsPerSeq, nFeatures): X = np.random.random((nSamples, nTimestepsPerSeq, nFeatures)) #make feature 1 categorical: [0,1,2] X[:,:,0] = np.random.randint(0,3, X[:,:,0].shape) #make feature 2 categorical: [-1, 0,1] X[:,:,1] = np.random.randint(-1,2, X[:,:,1].shape) #shift feature 3 by a constant X[:,:,2] = X[:,:,2] + 2 #calc output Y = np.zeros((1, nSamples)) #combine features and introduce non-linearity Y = X[:,-1,0]*np.mean(X[:,-1,3]) + X[:,-1,2]*np.mean(X[:,-1,4]) + \ (X[:,-1,0]*X[:,-1,1]*np.mean(X[:,-1,2]))**2 #add uniform noise Y = Y*np.random.uniform(0.95,1.05,size=Y.shape) #reshape for scaler instance: # ValueError: Expected 2D array, got 1D array instead: # array=[ 1.27764489 27.56604355 1.39317709 ..., 1.57210734 8.18834281 # 1.66174279]. # Reshape your data either using array.reshape(-1, 1) if your data has a single fe # ature or array.reshape(1, -1) if it contains a single sample. Y = Y.reshape((-1,1)) return X,Y Xtrain,Ytrain = generate_examples(5000, nTimestepsPerSeq, nFeatures) Xval,Yval = generate_examples(5000, nTimestepsPerSeq, nFeatures) Xtest,Ytest = generate_examples(20, nTimestepsPerSeq, nFeatures) #scale input data for i in range(0,nFeatures): #scaler = StandardScaler() scaler = MinMaxScaler() scaler = scaler.fit(Xtrain[:,:,i]) Xtrain[:,:,i] = scaler.transform(Xtrain[:,:,i]) Xval[:,:,i] = scaler.transform(Xval[:,:,i]) Xtest[:,:,i] = scaler.transform(Xtest[:,:,i]) targetScaler = MinMaxScaler() targetScaler = targetScaler.fit(Ytrain) #transform target Ytrain = targetScaler.transform(Ytrain) Yval = targetScaler.transform(Yval) Ytest = targetScaler.transform(Ytest) # defining the LSTM model model = Sequential() model.add(LSTM(200, input_shape=(Xtrain.shape, Xtrain.shape), return_sequences=True)) model.add(LSTM(200)) model.add(Dense(1)) model.compile(loss='mse', optimizer='adam', metrics=['acc']) # fitting the model history = model.fit(Xtrain, Ytrain, epochs=nEpochs, batch_size=50, validation_data=(Xval, Yval), shuffle=True, verbose=2) #test model yhat = model.predict(Xtest) print("pediction vs truth:") for i in range(0,10): print(yhat[i], Ytest[i]) # summarize history for loss plt.subplot(1,1,1) plt.plot(history.history['loss'], '.-') plt.plot(history.history['val_loss'], '.-') plt.ylabel('loss') plt.xlabel('epoch') plt.legend(['train', 'validation'], loc='upper right') plt.show()
Edit: I added some output of an experiment:
Epoch 1/12 - 6s - loss: 0.1056 - acc: 2.0000e-04 - val_loss: 0.0680 - val_acc: 0.0000e+00 Epoch 2/12 ... Epoch 11/12 - 4s - loss: 0.0033 - acc: 4.0000e-04 - val_loss: 0.0020 - val_acc: 0.0000e+00 Epoch 12/12 - 4s - loss: 0.0016 - acc: 4.0000e-04 - val_loss: 0.0016 - val_acc: 0.0000e+00 pediction vs truth: [ 0.25022525] [ 0.25465108] [ 0.98761547] [ 0.91661543] [ 1.06177747] [ 0.95979166] [ 0.0835482] [ 0.03742919] [ 0.02432941] [ 0.01149685] [ 0.00915699] [ 0.00887351] [ 0.2765356] [ 0.27340488] [ 0.02941256] [ 0.01685951] [-0.01059875] [ 0.00157809] [ 0.04762106] [ 0.01983566]