Scenario 1. Let's say I have 3 response series (Y1,Y2,Y3) and one exogenous variable (X1). Thus, I am training LSTM on batches of form [Y1_,Y2_,Y3_,X1_] to predict [Y1*,Y2*,Y3*], where _ indicates a corresponding slice and * is (t+1) observation.
Scenario 2. Now suppose I train a separate model with responses (Y3,Y2,Y1) feeding [Y3_,Y2_,Y1_,X1_] to predict [Y3*,Y2*,Y1*], i.e. literally exchanged columns Y1 and Y3. The results (performance on the test set) are different now. In particular, Y3 and Y1 are predicted better and worse respectively in comparison with scenario 1.
I have 'my own' implementation of LSTM in tensorflow, but I also checked this in keras to isolate the anomaly and have the same problem there. The way I create those slices before feeding into the model seems correct/consistent. Is this a bug?