I am trying to code a recurrent neural network (LSTM) to create music in python and was considering using multiple features instead of just the note pitch as an input into the network. Initially I had just the note pitch so it was fed into the network by one-hot encoding it. The other two features I want to add are the note duration and the offset between the notes. How should the input vector be organised so that all the data is fed through the network?

I have tried combining all of the data into a long vector with all 3 features one-hot encoded and then concatenated but this caused the output to become 'NaN'. Any help would be appreciated.

Link a gist of my code:


  • $\begingroup$ which deep learning library are you using? tensor-flow or pytorch? Also for this type of question, please include your code in the question. $\endgroup$ – Louis T Feb 9 '19 at 9:56
  • $\begingroup$ @LouisT I’m using a numpy implementation of an LSTM should I just put the code for forward propagation? $\endgroup$ – treutm Feb 9 '19 at 9:57
  • $\begingroup$ The NaN can occur at backdrop as well. If your code is too long, put it in a gist on GitHub and include a link to it. $\endgroup$ – Louis T Feb 9 '19 at 10:02
  • $\begingroup$ @LouisT Just added the link to the gist. Just checking, can you view it? (haven't made a gist before) $\endgroup$ – treutm Feb 9 '19 at 10:41
  • $\begingroup$ congrats on your first gist :) and yes, I can see the gist. Can you try to modify this code such that someone can simply copy the code and run it on their machine to reproduce the NaN you are seeing? You might need to save the output your read_mini in another gist. $\endgroup$ – Louis T Feb 9 '19 at 11:12

A common way to input several features to an LSTM (or any RNN) is, as you did, to concatenate them in a vector. I suspect your NaN are related to a different issue in the code, and I recommend you to debug it and see when and why it happens.

A different way of combining several features is using embeddings for each feature and combine them via concatenation. For example, for the note pitch, you have an embedding for each note, for the note duration, the same, and so on. The input to the RNN would be a concatenation of that.

A third option would be to have an embedding to all possible combinations of the embeddings. That would be an embedding for each triplet of note pitch, note duration and offset. This representation can be also powerful.

My recommendation is that you start with the easiest to implement, and if the result is not satisfactory in terms of model performance, think about the next one.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.