Running an LSTM with Music Data

Question

I'm working on a project for a class where I'm trying to create an algorithm that learns music and creates its own music.

I'm having trouble on how to set up the data for it to be inputted into the LSTM.

A single training example consists of a chord that is a vector of binary values based on what keys are pressed in MIDI form (indices 0-127), a value that denotes duration of the note, beat strength, numerator of the time signature, and denominator of the time signature, and the key signature represented by the number of flats

So one example might look like

$$\left[ \begin{array}{c} {0} \\ {1} \\{0} \\{1} \\{\vdots} \\ {1} \\ {0} \\ {4} \\ {3} \\ {4} \\ {4} \\ {-2} \end{array} \right]$$

The result is a 132x1 vector

I was having trouble conceptualizing how to input this data type into an LSTM. Doing a linear output would not make that much sense, but I don't think I can directly one-hot this vector either.

Answer 1

You should ask yourself - are you teaching an algorithm to play chords or to play music? In addition, what are you trying to predict here?

It seems to me that you need to create input data that is a series of chords and your label is the next chord in the tune. So you should design a neural network that takes in a series of chords and can tell you the next chord in the sequence, add that back to the input sequence and pick the next chord, add that back to the input sequence, etc, etc. Next thing you know, you have a neural network that can play music.

Answer 2

If you are using Tensorflow, then create the input tensor of the dimension as given below :

input_data = tf.placeholder(tf.float32, [batch_size, timesteps, input_size], name='inputs')

Answer 3

To input this type of data into an LSTM, you can use an embedding layer, which is commonly used in natural language processing (NLP) tasks to represent discrete input tokens as continuous vectors.

In your case, you can create an embedding layer for each of the categorical variables in your input vector (i.e., the chord, duration, beat strength, numerator of the time signature, denominator of the time signature, and key signature). Each embedding layer will map the discrete variable to a continuous vector representation.

For the chord input, you can represent it as a one-hot encoded vector of length 128 (corresponding to the MIDI note numbers). You can then pass this one-hot encoded vector through an embedding layer with a specified embedding size (e.g., 32) to get a 32-dimensional vector representation of the chord.

For the continuous variables (i.e., duration, beat strength, numerator of the time signature, denominator of the time signature, and key signature), you can pass them through a linear layer to get a continuous vector representation.

Once you have obtained the continuous vector representations for each input variable, you can concatenate them into a single vector and pass it through the LSTM.

The output of the LSTM can then be passed through a linear layer to obtain the predicted chord for the next time step.