My Keras bidirectional LSTM model is giving terrible predictions

Question

I am trying to predict velocity (dynamics) values for notes that make up a piece of music using a bi-directional LSTM, following this blog post pretty closely: http://imanmalik.com/cs/2017/06/05/neural-style.html

I think I have set up everything correctly, but when predicting my model seems to produce only gibberish. Here's a plot of a prediction done on a validation sample (y-axis is always time and x-axis is, for I., a note activation array; for II. the predicted velocities and for III. the true, expected velocities). As you can see, the predicted velocity values (in the middle graph) are identical for all time steps (save for a few in the very beginning and at the very end, but that's hard to see in the plot):

For this particular prediction I had trained my model with 44 batches of 4 samples each and for 1 epoch, but I have also tried training it for 20 epochs and with different batch sizes and it doesn't seem to give better results.

Using TensorFlow backend.
Loading numpy data ...
173 train sequences
10 test sequences
x_train shape: (173,)
y_train shape: (173,)
Setting up model ...
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
bidirectional_1 (Bidirection (4, None, 128)            123392
_________________________________________________________________
bidirectional_2 (Bidirection (4, None, 128)            98816
_________________________________________________________________
bidirectional_3 (Bidirection (4, None, 128)            98816
_________________________________________________________________
time_distributed_1 (TimeDist (4, None, 88)             11352
=================================================================
Total params: 332,376
Trainable params: 332,376
Non-trainable params: 0
_________________________________________________________________
None
Training model ...
Epoch 1/1
2018-03-10 14:47:27.650226: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.2 AVX
44/44 [==============================] - 2145s 49s/step - loss: 0.0783 - mean_squared_error: 0.0783
Saving model ...
Loss and metrics: [0.002714238129556179, 0.0027141105383634567]

So, in other words, I end up with a validation MSE of 0.0027 after 1 epoch. Here is the code for my Keras model:

number_of_notes = 88
input_size = number_of_notes * 2 # 88 notes * 2 states (pressed, sustained).
output_size = number_of_notes # 88 notes.

units = 64
dropout = 0.2 # Drop 20% of units for linear transformation of inputs.

model = Sequential()
model.add(Bidirectional(LSTM(units, return_sequences=True, dropout=dropout),
                        input_shape=(None,input_size),
                        batch_input_shape=(args.batch_size,None,input_size)))
model.add(Bidirectional(LSTM(units, return_sequences=True, dropout=dropout)))
model.add(Bidirectional(LSTM(units, return_sequences=True, dropout=dropout)))
model.add(TimeDistributed(Dense(output_size, activation='sigmoid')))
model.compile(loss='mse', optimizer=Adam(lr=0.001, clipnorm=10), metrics=['mse'])

print('Training model ...')

number_of_train_batches = np.ceil(len(x_train)/float(args.batch_size))
model.fit_generator(batch_generator(x_train, y_train),
                    steps_per_epoch=number_of_train_batches,
                    epochs=args.epochs)

Here is how I predict:

prediction_data = np.load(path)

# Copy prediction input N times to create a batch of the right size.
tiled = np.tile(prediction_data, [args.batch_size,1,1])

raw_prediction = model.predict(tiled, batch_size=args.batch_size)[0]
prediction = (raw_prediction * 127).astype(int) # Float -> MIDI velocity.

So what I am asking is, where am I likely to be going wrong with my model? Which parameters should I try changing in order to get better results? Or how can I change my model or training set-up to get better results? Etc. Any and all help is appreciated!

Answer 1

I think you cannot use a bi-directional LSTM for prediction, because of the time dimension of the music. I mean the backwards layer has to predict the latest value first and only after predicting it sees the sequence which gives the context- This is like you watch a reversed movie and yo have to guess how the first frame looks like without knowing the rest of it.

Also, check out stylenet of the imanmalik model. The interpretation lstm which does the prediction is unidirectional. Only the genre net units are bidirectional, but they get the predicted value already as input.

Answer 2

The original blog post mentions that the interpretation layer reduces the overall parameter size of the model, although I couldn't figure out by how much. It doesn't look like your network has that, and it feels really large; having only 173 training examples, even if their dimensionality is large, feels like a small amount for a network with 300k+ examples. Can you try a smaller version of the network and see if it can train on the data you have?

Also, can you explain the step where you're copying the input data $N$ times?

Answer 3

I ended up solving this issue thanks to some excellent advice by a colleague of mine, which amounted to this:

Pare the model down to the bare essentials and see if you get sensible output then. If you do, add elements / layers one by one and see where it starts to go wrong.

Here I believe the issue was with the Sigmoid activation of the final layer. The model I have now is working well, and looks like this:

model = Sequential()
model.add(Bidirectional(LSTM(output_size, activation='relu', return_sequences=True, dropout=dropout),
                        merge_mode='sum',
                        input_shape=(None, input_size),
                        batch_input_shape=(batch_size, None, input_size)))
model.add(Bidirectional(LSTM(output_size, activation='relu', return_sequences=True,
                             dropout=dropout), merge_mode='sum'))
model.add(Bidirectional(LSTM(output_size, activation='relu', return_sequences=True,
                             dropout=dropout), merge_mode='sum'))
model.compile(loss='mse', optimizer=Adam(
    lr=0.001, clipnorm=1), metrics=['mse'])