Value Error: Operands could not be broadcast together with shapes - LSTM

Question

I am trying an LSTM model using tensorflow following this tutorial .

I am having trouble understanding why am I getting an error in my test set when I try to invert scaling for forecast (line 86 in the tutorial).

After loading the dataset and created featured, following is what I did for reframed dataset,

# split into train and test sets
values = reframed.values
split_point = len(reframed)- 168

train = values[ : split_point, :]
test = values[split_point: , :]

# split into input and outputs
train_X, train_y = train[:, :-1], train[:, -1]
test_X, test_y = test[:, :-1], test[:, -1]

# reshape input to be 3D [samples, timesteps, features]
train_X = train_X.reshape((train_X.shape[0], 1, train_X.shape[1]))
test_X = test_X.reshape((test_X.shape[0], 1, test_X.shape[1]))

print(train_X.shape, train_y.shape, test_X.shape, test_y.shape)
>> (2399, 1, 39) (2399,) (168, 1, 39) (168,)


# design network
model = Sequential()
model.add(LSTM(50, input_shape=(train_X.shape[1], train_X.shape[2])))
model.add(Dense(1))
model.compile(loss='mae', optimizer='adam')


# fit network
history = model.fit(train_X, train_y, epochs=50, batch_size=72, validation_data=(test_X, test_y), verbose=2, shuffle=False)

# make a prediction
yhat = model.predict(test_X)
test_X = test_X.reshape((test_X.shape[0], test_X.shape[2]))

inv_yhat = np.concatenate((yhat, test_X[:, 1:]), axis=1)

After this step, the following code that has scaler.invert_transform throws the error.

inv_yhat = scaler.inverse_transform(inv_yhat)

ValueError: operands could not be broadcast together with shapes (168,39) (41,) (168,39)

Shape,

test_X.shape
>> (168, 1, 39)

yhat.shape
>>(168,1)

inv_yhat.shape
>> (168,39)

Any help or sugeestions would be appreciated!

Answer 1

The answer that OP provided is correct, yet I would like to elaborate a little more on it, in an attempt to shed more light.

First of all, you have to understand what is performed by the call

reframed = series_to_supervised(scaled, 1, 1)

in line 50.

Shortly, let us assume that the input has n samples of m features (variables). This means that the shape of the scaled dataframe is (n) x (m). The particular call to series_to_supervised() in line 50 implies that the scaled dataframe will be reshaped into a (n-1) x (2m) dataframe, to include the current and the 1-step look-back values, after dropping the row of NaN values due to the shift operation inside the body of series_to_supervised(). Most likely, your error is caused by line 52:

reframed.drop(reframed.columns[[9,10,11,12,13,14,15]], axis=1, inplace=True)

because you are probably dropping an arbitrary number of columns. The code is structured so that it forecasts one time series based on the past values of multiple time series (including the signal of interest). Therefore, you have to drop (m-1) columns, so that in the end all m past values persist, along with the current value of the signal of interest that will be used as ground truth. This means that the shape of reframed should be (n-1) x (m+1) after line 52. If not, you probably get that error with the dimension mismatch when reverting the scaling.

At this point I would like to emphasize on the fact that you should be very careful with logical errors when dropping columns, because you want to drop all but the signal of interest in line 52. If you keep any other column the code will still run, but you will certainly get logical artifacts.

Moreover, if you drop any column other than the first one (at index 0), you have to modify the concatenation in lines 85 & 90 accordingly, otherwise you will also get logical artifacts.

Answer 2

I thought of posting as an answer according to the suggestion by @Seteve Kallestad in case of someone come across this issue.

The answer is: The shape of the data must be the same when inverting the scale as when it was originally scaled.

I was making an error in my data preparation. Initially, when I scaled the data my test set had a shape of (168,39).

As it mentions in the error

shapes (168,39) (41,) (168,39)

I had 41 columns and it should have been 39.

The reason is shape of yhat is (168,1) and shape of test_X[:, 1:] is 168,38 as it starts from column number 1. When they are concatenated it becomes (168,39) which is exactly how it was when it was scaled.