How to input LSTM output to MLP with concatenate?

Question

I am having a training data set for a time-series dataset like below where my target variable is var1(t) which is the value of var 1 at time=t.

import numpy as np
import pandas as pd
train_df = pd.DataFrame(np.random.randint(0,100,size=(100, 16)))
train_df.columns=['var1(t-3)','var2(t-3)','var3(t-3)','var4(t-3)','var1(t-2)','var2(t-2)','var3(t-2)','var4(t-2)','var1(t-1)','var2(t-1)','var3(t-1)','var4(t-1)','var1(t)','var2(t)','var3(t)','var4(t)']
train_X=train_df.drop(['var1(t)'],axis=1)
train_y=train_df[['var1(t)']]

I am inputting the LSTM network with the past 3 timesteps (t-3) to (t-1) of all 4 variables and then feed the output of the LSTM with the current timestep value of the var2,var3,var4 with an MLP with functional API in Keras.

So I prepared the inputs for the LSTM and MLP like below :

#subset the 3 previous timesteps of the 4 variables for the time sries part 
train_X_LSTM=train_X[train_X.columns[:12]].reset_index(drop=True).values
#target is always var1(t)
train_y_LSTM=train_y.values
#take the current timestep fatures which are var2,var3,var4 which are realised at t=t
train_X_MLP=train_X[train_X.columns[-3:]].reset_index(drop=True).values
#target is always var1(t)
train_y_MLP=train_y.values

Then I tried the below network :

#lstm input shape
lstm_input = Input(shape=(train_X_LSTM.shape[0],train_X_LSTM.shape[1]))
#lstm units
hidden1 = LSTM(10)(lstm_input)
hidden2 = Dense(10, activation='relu')(hidden1)
#lstm output which will be predicted var1 at t=t
lstm_output = Dense(1, activation='sigmoid')(hidden2)
#mlp input with additonal 3 variables at t=t
mlp_input=Input(shape=(train_X_MLP.shape[0],train_X_MLP.shape[1]))
#combine the lstm output which is predicted var1 at t=t and key in var2,var3,var4 at t=t
x = concatenate([lstm_output, mlp_input], axis=-1)
#mlp model output which is predicted var1 at t=t
mlp_out = Dense(1, activation='relu')(x)
#final output of combined model which is predicted var1 at t=t
model = Model(inputs=[lstm_input, mlp_input],outputs=mlp_out)
#compile the model
model.compile(loss='mae', optimizer='adam')
#fit the model
model.fit(x_train, y_train, batch_size=64, epochs=10, validation_split=0.2)

This throws the error of

ValueError: A Concatenate layer requires inputs with matching shapes except for the concat axis. Got inputs shapes: [(None, 1), (None, 100, 3)]

which shows I am not combining them correctly. Any help is appreciated!

Answer 1

In Keras, there is no need to give the number of samples in your training data to your model. Also, the way you defined your LSTM means that you consider each data sample as a timestep for the LSTM, instead of your t-1, t-2 and t-3 values.

So you can remove train_X_LSTM.shape[0] in your Input layers, and give X=[train_X_LSTM, train_X_MLP] and y=train_y_LSTM to model.fit so it matches what your model expects.

#lstm input shape
lstm_input = Input(shape=(train_X_LSTM.shape[1], 1))
#lstm units
hidden1 = LSTM(10)(lstm_input)
hidden2 = Dense(10, activation='relu')(hidden1)
#lstm output which will be predicted var1 at t=t
lstm_output = Dense(1, activation='sigmoid')(hidden2)
#mlp input with additonal 3 variables at t=t
mlp_input=Input(shape=(train_X_MLP.shape[1]))
#combine the lstm output which is predicted var1 at t=t and key in var2,var3,var4 at t=t
x = Concatenate()([lstm_output, mlp_input])
#mlp model output which is predicted var1 at t=t
mlp_out = Dense(1, activation='relu')(x)
#final output of combined model which is predicted var1 at t=t
model = Model(inputs=[lstm_input, mlp_input],outputs=mlp_out)
#compile the model
model.compile(loss='mae', optimizer='adam')
#fit the model
model.fit([train_X_LSTM, train_X_MLP], train_y_LSTM, batch_size=64, epochs=10, validation_split=0.2)

In addition, from my understanding, you want to input (var1, var2, var3, var4) values vector at timesteps t-3, t-2, t-1 to the LSTM. In that case, you should also reshape the data you feed to the LSTM

train_X_LSTM=train_X_LSTM.reshape(-1, 3, 4)

This way, you will feed a sequence of 3 vectors containing (var1, var2, var3, var4) at the corresponding past timesteps. Then, update the LSTM Input layer

lstm_input = Input(shape=train_X_LSTM.shape[1:])

Full code

import numpy as np
import pandas as pd
train_df = pd.DataFrame(np.random.randint(0,100,size=(100, 16)))
train_df.columns=['var1(t-3)','var2(t-3)','var3(t-3)','var4(t-3)','var1(t-2)','var2(t-2)','var3(t-2)','var4(t-2)','var1(t-1)','var2(t-1)','var3(t-1)','var4(t-1)','var1(t)','var2(t)','var3(t)','var4(t)']
train_X=train_df.drop(['var1(t)'],axis=1)
train_y=train_df[['var1(t)']]
#subset the 3 previous timesteps of the 4 variables for the time sries part 
train_X_LSTM=train_X[train_X.columns[:12]].reset_index(drop=True).values
train_X_LSTM=train_X_LSTM.reshape(-1, 3, 4)
#target is always var1(t)
train_y_LSTM=train_y.values
#take the current timestep fatures which are var2,var3,var4 which are realised at t=t
train_X_MLP=train_X[train_X.columns[-3:]].reset_index(drop=True).values
#target is always var1(t)
train_y_MLP=train_y.values

from tensorflow.keras.layers import Input, LSTM, Dense, Concatenate
from tensorflow.keras.models import Model
#lstm input shape
lstm_input = Input(shape=train_X_LSTM.shape[1:])
#lstm units
hidden1 = LSTM(10)(lstm_input)
hidden2 = Dense(10, activation='relu')(hidden1)
#lstm output which will be predicted var1 at t=t
lstm_output = Dense(1, activation='sigmoid')(hidden2)
#mlp input with additonal 3 variables at t=t
mlp_input=Input(shape=(train_X_MLP.shape[1]))
#combine the lstm output which is predicted var1 at t=t and key in var2,var3,var4 at t=t
x = Concatenate()([lstm_output, mlp_input])
#mlp model output which is predicted var1 at t=t
mlp_out = Dense(1, activation='relu')(x)
#final output of combined model which is predicted var1 at t=t
model = Model(inputs=[lstm_input, mlp_input],outputs=mlp_out)
#compile the model
model.compile(loss='mae', optimizer='adam')
#fit the model
model.fit([train_X_LSTM, train_X_MLP], train_y_LSTM, batch_size=64, epochs=10, validation_split=0.2)