Customize loss function for Music Generation LSTM (?)

Question

I have to carry out a Music Generation project for a Deep Learning course I have this semester and I am using Pytorch. The dataset is songs in midi format and I use the python library mido to extract the data out of every song. The data in every midi song are organized as a series of discrete event messages.

I plan to have my input of the LSTM for each time-step as a vectorized event of three variables:

x_t = [note, velocity, timestamp]

note: categorical variable, since it can take only one integer value in the range [0:127] which corresponds to a pitch.

velocity: I think it is categorical too, as it also takes one integer value in the range [0:127] which corresponds to the force with which the note is played.

timestamp: the continuous variable that corresponds to the number of seconds (mostly milliseconds) or ticks that have passed between the current and previous events.

My question is since I have a combination of categorical and continuous variables to predict:

• Is it ok to one-hot-encode note and velocity and keep timestamp as is?
• Should I further standardize timestamp?
• Can I use a customized loss function (sum of cross-entropy loss for note/velocity plus mean squared error for timestamp), or it would mess up my model?

Please, I feel a bit lost and need your help/suggestions.

Thank you!!!

Answer 1

That's not a good way to represent symbolic monophonic music.

The biggest problem is that time should be modelled as a discrete and not a continuous variable. The reason for that is that music with notes that are off the beat sounds like crap. E.g if the tempo indicates that one beat is 250 ms long and one beat randomly is only 200 ms long it will sound very bad.

Instead, fix the tempo so that, say one quarter note is 125 ms long and have your time variable be an integer factor of that. You could cut the range to [1, 16] so that the shortest note is 125 ms long and the longest 2 000 ms (125 * 16).

Then I would skip the velocity because it is very hard to model. I don't think even state-of-the-art models incorporates it and most music sound fine even without it.

For pitches, I would cut down their range to three or four octaves. Most MIDI songs doesn't use more scales than that and it makes it much easier for your model to generalize.

So with four octaves, you are left with two variables $t \in [1,16]$ for time and $p \in [1,48]$ for pitch. You can either assume these are independent (which they aren't, but the generated music might still sound good!) and model them separately, giving you a sequence of pairs of discreete values: $$ (t_1, p_1), (t_2, p_2), ... $$ Or you could multiply $t$ and $p$ so that each input to the network is an integer in the range [1,16*48]: $$ t_1p_1, t_2p_2, ... $$ You could also interleave the input so that every other item in the sequence contains timing and pitch information: $$ t_1, p_1, t_2, p_2, ... $$ I don't know which way is the best - you'll have to experimetn.

I'm fairly certain it won't matter which loss function you choose. But I would go with crossentropy because it usually works well with discrete variables.