4
$\begingroup$

I am trying an LSTM model using tensorflow following this tutorial .

I am having trouble understanding why am I getting an error in my test set when I try to invert scaling for forecast (line 86 in the tutorial).

After loading the dataset and created featured, following is what I did for reframed dataset,

# split into train and test sets
values = reframed.values
split_point = len(reframed)- 168

train = values[ : split_point, :]
test = values[split_point: , :]

# split into input and outputs
train_X, train_y = train[:, :-1], train[:, -1]
test_X, test_y = test[:, :-1], test[:, -1]

# reshape input to be 3D [samples, timesteps, features]
train_X = train_X.reshape((train_X.shape[0], 1, train_X.shape[1]))
test_X = test_X.reshape((test_X.shape[0], 1, test_X.shape[1]))

print(train_X.shape, train_y.shape, test_X.shape, test_y.shape)
>> (2399, 1, 39) (2399,) (168, 1, 39) (168,)


# design network
model = Sequential()
model.add(LSTM(50, input_shape=(train_X.shape[1], train_X.shape[2])))
model.add(Dense(1))
model.compile(loss='mae', optimizer='adam')


# fit network
history = model.fit(train_X, train_y, epochs=50, batch_size=72, validation_data=(test_X, test_y), verbose=2, shuffle=False)

# make a prediction
yhat = model.predict(test_X)
test_X = test_X.reshape((test_X.shape[0], test_X.shape[2]))

inv_yhat = np.concatenate((yhat, test_X[:, 1:]), axis=1)

After this step, the following code that has scaler.invert_transform throws the error.

inv_yhat = scaler.inverse_transform(inv_yhat)

ValueError: operands could not be broadcast together with shapes (168,39) (41,) (168,39)

Shape,

test_X.shape
>> (168, 1, 39)

yhat.shape
>>(168,1)

inv_yhat.shape
>> (168,39)

Any help or sugeestions would be appreciated!

$\endgroup$
  • 1
    $\begingroup$ Better to include the code sample (or at least part of it) here. External links disappear over time. $\endgroup$ – Steve Kallestad Aug 22 '17 at 20:02
  • $\begingroup$ @SteveKallestad thank you for the response. sure, I can include it. It will be really long about 90 lines of code. $\endgroup$ – i.n.n.m Aug 22 '17 at 20:05
  • 1
    $\begingroup$ The error occurs because of the variant in the shapes. I don't have time to dig into it now, but I would have a look at the data and see if it hasn't changed or wasn't prepared correctly. $\endgroup$ – Steve Kallestad Aug 22 '17 at 20:07
  • $\begingroup$ @SteveKallestad i agree with you, if you have time, there's a question about this on the tutorial and the author asked that person to check the data preparation. I went back and checked mine, print(train_X.shape, train_y.shape, test_X.shape, test_y.shape) and I got (2399, 1, 39) (2399,) (168, 1, 39) (168,) like this. $\endgroup$ – i.n.n.m Aug 22 '17 at 20:11
  • 1
    $\begingroup$ @SteveKallestad thank you for your suggestion, I could find the error and it works perfect now. I posted my mistake as an answer, in case if someone comes across this issue! $\endgroup$ – i.n.n.m Aug 23 '17 at 21:54
7
$\begingroup$

The answer the OP provided is correct, yet I would like to elaborate a little more on it, in an attempt to shed more light.

First of all, you have to understand what is performed by the call

reframed = series_to_supervised(scaled, 1, 1)

in line 50.

Shortly, let us assume that the input has n samples of m features (variables). This means that the shape of the scaled DataFrame is (n) x (m). The particular call to series_to_supervised() at line 50 implies that the scaled DataFrame will be reshaped into a (n-1) x (2m) DataFrame, to include the current and the 1-step look-back values, after dropping the row of NaN values that occurs due to the shift operation inside the body of series_to_supervised(). The most probable cause of your error relies on line 52:

reframed.drop(reframed.columns[[9,10,11,12,13,14,15]], axis=1, inplace=True)

because you are probably dropping an arbitrary number of columns. The code is structured so that it forecasts one time series based on the past values of multiple time series (including the signal of interest). Therefore, you have to drop (m-1) columns, so that in the end all m past values persist, along with the current value of the signal of interest that will be used as ground truth, therefore the shape of reframed should be (n-1) x (m+1) after line 52. If not, you probably get that error with the dimension mismatch when reverting the scaling.

At this point I would like to emphasize on the fact that you should be very careful with logical errors when dropping columns, because you want to drop all but the signal of interest in line 52. If you keep any other column, the code will run but you will obviously get artifacts.

Needless to say, if you drop any column other than the first one (at index 0), you have to modify the concatenation in lines 85 & 90 accordingly, otherwise you will get logical artifacts.

Finally, as a general tip for troubleshooting such problems without debuggers or redundant intermediate "prints", I would highly recommend you using a code editor such as Spyder3 editor, because it provides you with a clean variable explorer that shows the shapes of all variables after each code execution.

$\endgroup$
2
$\begingroup$

I thought of posting as an answer according to the suggestion by @Seteve Kallestad in case of someone come across this issue.

The answer is: The shape of the data must be the same when inverting the scale as when it was originally scaled.

I was making an error in my data preparation. Initially, when I scaled the data my test set had a shape of (168,39).

As it mentions in the error

shapes (168,39) (41,) (168,39)

I had 41 columns and it should have been 39.

The reason is shape of yhat is (168,1) and shape of test_X[:, 1:] is 168,38 as it starts from column number 1. When they are concatenated it becomes (168,39) which is exactly how it was when it was scaled.

$\endgroup$
  • $\begingroup$ I am still struggling with the same error and I can't seem to understand the answer you posted. Can you please let me know how you managed to bypass this error?? $\endgroup$ – Effa Jun 12 '18 at 5:18
  • $\begingroup$ Hi Effa. Are you using the exact same tutorial code and you just changed the dataset? $\endgroup$ – pcko1 Jun 12 '18 at 7:48
  • 1
    $\begingroup$ @Effa Trick is to always check dimensionality of your data frames (training and test sets). #pcko1 answer explains well. $\endgroup$ – i.n.n.m Jun 12 '18 at 14:13
  • $\begingroup$ If you have more or less features than in the exampe, then in the second version at the bottom of the tutorial, replace '-7' with '(n_features-1)' in lines 88 and 93 i.e. inv_yhat = concatenate((yhat, test_X[:, -(n_features-1):]), axis=1). This corrects the dimensions. $\endgroup$ – intotecho Feb 5 at 3:40

protected by Community Oct 23 '18 at 13:38

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.