I am trying to predict velocity (dynamics) values for notes that make up a piece of music using a bi-directional LSTM, following this blog post pretty closely: http://imanmalik.com/cs/2017/06/05/neural-style.html
I think I have set up everything correctly, but when predicting my model seems to produce only gibberish. Here's a plot of a prediction done on a validation sample (y-axis is always time and x-axis is, for I., a note activation array; for II. the predicted velocities and for III. the true, expected velocities). As you can see, the predicted velocity values (in the middle graph) are identical for all time steps (save for a few in the very beginning and at the very end, but that's hard to see in the plot):
For this particular prediction I had trained my model with 44 batches of 4 samples each and for 1 epoch, but I have also tried training it for 20 epochs and with different batch sizes and it doesn't seem to give better results.
Using TensorFlow backend. Loading numpy data ... 173 train sequences 10 test sequences x_train shape: (173,) y_train shape: (173,) Setting up model ... _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= bidirectional_1 (Bidirection (4, None, 128) 123392 _________________________________________________________________ bidirectional_2 (Bidirection (4, None, 128) 98816 _________________________________________________________________ bidirectional_3 (Bidirection (4, None, 128) 98816 _________________________________________________________________ time_distributed_1 (TimeDist (4, None, 88) 11352 ================================================================= Total params: 332,376 Trainable params: 332,376 Non-trainable params: 0 _________________________________________________________________ None Training model ... Epoch 1/1 2018-03-10 14:47:27.650226: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.2 AVX 44/44 [==============================] - 2145s 49s/step - loss: 0.0783 - mean_squared_error: 0.0783 Saving model ... Loss and metrics: [0.002714238129556179, 0.0027141105383634567]
So, in other words, I end up with a validation MSE of 0.0027 after 1 epoch. Here is the code for my Keras model:
number_of_notes = 88 input_size = number_of_notes * 2 # 88 notes * 2 states (pressed, sustained). output_size = number_of_notes # 88 notes. units = 64 dropout = 0.2 # Drop 20% of units for linear transformation of inputs. model = Sequential() model.add(Bidirectional(LSTM(units, return_sequences=True, dropout=dropout), input_shape=(None,input_size), batch_input_shape=(args.batch_size,None,input_size))) model.add(Bidirectional(LSTM(units, return_sequences=True, dropout=dropout))) model.add(Bidirectional(LSTM(units, return_sequences=True, dropout=dropout))) model.add(TimeDistributed(Dense(output_size, activation='sigmoid'))) model.compile(loss='mse', optimizer=Adam(lr=0.001, clipnorm=10), metrics=['mse']) print('Training model ...') number_of_train_batches = np.ceil(len(x_train)/float(args.batch_size)) model.fit_generator(batch_generator(x_train, y_train), steps_per_epoch=number_of_train_batches, epochs=args.epochs)
Here is how I predict:
prediction_data = np.load(path) # Copy prediction input N times to create a batch of the right size. tiled = np.tile(prediction_data, [args.batch_size,1,1]) raw_prediction = model.predict(tiled, batch_size=args.batch_size) prediction = (raw_prediction * 127).astype(int) # Float -> MIDI velocity.
So what I am asking is, where am I likely to be going wrong with my model? Which parameters should I try changing in order to get better results? Or how can I change my model or training set-up to get better results? Etc. Any and all help is appreciated!