I am planning on making an AI song composer that would take in a bunch of songs of one instrument, extract musical notes (like ABCDEFG) and certain features from the sound wave, preform machine learning (most likely through recurrent neural networks), and output a sequence of ABCDEFG notes (aka generate its own songs / music).

I think that this would be an unsupervised learning problem, but I am not really sure.

I figured that I would use recurrent neural networks, but I have a few questions on how to approach this:
- What features from the sound wave I should extract so that the output music is melodious?
- Is it possible, with recurrent neural networks, to output a vector of sequenced musical notes (ABCDEF)?
- Any smart way I can feed in the features of the soundwaves as well as sequence of musical notes?

  • 3
    $\begingroup$ I think you are grossly underestimating the challenge of AI music composition, and you are trying to take on music recognition at the same time. I also think you (and many other people) have an inflated opinion of what ML (including recurrent neural networks) can do. $\endgroup$ Jul 15 '15 at 5:27
  • $\begingroup$ Have you considered how you will deal with instruments that produce cords, such as piano, guitar and other stringed instruments? $\endgroup$ Jul 15 '15 at 10:10
  • $\begingroup$ Deep Dreaming for audio? $\endgroup$ Jul 15 '15 at 12:39
  • 1
    $\begingroup$ This is such a tough problem that you'll have to do a lot of paper sighting. I don't think it's possible to sum everything up that you need to know in a few paragraphs (since there are no established best practices) $\endgroup$
    – runDOSrun
    Jul 15 '15 at 17:22
  • $\begingroup$ Great...Negative comments might encourage you more to investigate. I think Restricted Boltzmann machines work well with nonlinear data.To extract features you need to read papers on speech recognition. Good luck $\endgroup$
    – Mary
    Jul 20 '17 at 13:31

First off, ignore the haters. I started working on ML in Music a long time ago and got several degrees using that work. When I started I was asking people the same kind of questions you are. It is a fascinating field and there is always room for someone new. We all have to start somewhere.

The areas of study you are inquiring about are Music Information Retrieval (Wiki Link) and Computer Music (Wiki Link) . You have made a good choice in narrowing your problem to a single instrument (monophonic music) as polyphonic music increases the difficulty greatly.

You're trying to solve two problems really:

1) Automatic Transcription of Monophonic Music (More Readings) which is the problem of extracting the notes from a single instrument musical piece.

2) Algorithmic Composition (More Readings) which is the problem of generating new music using a corpus of transcribed music.

To answer your questions directly:

I think that this would be an unsupervised learning problem, but I am not really sure.

Since there are two learning problems here there are two answers. For the Automatic Transcription you will probably want to follow a supervised learning approach, where your classification are the notes you are trying to extract. For the Algorithmic Composition problem it can actually go either way. Some reading in both areas will clear this up a lot.

What features from the sound wave I should extract so that the output music is melodious?

There are a lot of features used commonly in MIR. @abhnj listed MFCC's in his answer but there are a lot more. Feature analysis in MIR takes place in several domains and there are features for each. Some Domains are:

  1. The Frequency Domain (these are the values we hear played through a speaker)
  2. The Spectral Domain (This domain is calculated via the Fourier function (Read about the Fast Fourier Transform) and can be transformed using several functions (Magnitude, Power, Log Magnitude, Log Power)
  3. The Peak Domain (A domain of amplitude and spectral peaks over the spectral domain)
  4. The Harmonic Domain

One of the first problems you will face is how to segment or "cut up" your music signal so that you can extract features. This is the problem of Segmentation (Some Readings) which is complex in itself. Once you have cut your sound source up you can apply various functions to your segments before extracting features from them. Some of these functions (called window functions) are the: Rectangular, Hamming, Hann, Bartlett, Triangular, Bartlett_hann, Blackman, and Blackman_harris.

Once you have your segments cut from your domain you can then extract features to represent those segments. Some of these will depend on the domain you selected. A few example of features are: Your normal statistical features (Mean, Variance, Skewness, etc.), ZCR, RMS, Spectral Centroid, Spectral Irregularity, Spectral Flatness, Spectral Tonality, Spectral Crest, Spectral Slope, Spectral Rolloff, Spectral Loudness, Spectral Pitch, Harmonic Odd Even Ratio, MFCC's and Bark Scale. There are many more but these are some good basics.

Is it possible, with recurrent neural networks, to output a vector of sequenced musical notes (ABCDEF)?

Yes it is. There have been several works to do this already. (Here are several readings)

Any smart way I can feed in the features of the soundwaves as well as sequence of musical notes?

The standard method is to use the explanation I made above (Domain, Segment, Feature Extract) etc. To save yourself some work I highly recommend starting with a MIR framework such as MARSYAS (Marsyas). They will provide you with all the basics of feature extraction. There are many frameworks so just find one that uses a language you are comfortable in.


I believe the question is, you want to learn from musical pieces and try to generate a tune from the trained instance. Lets see if I can set up a simple model to do this, and then you can extrapolate from there.

So, MFCC is a good feature when working with sound. You can use that to extract the features from lets say 1-2 second windows of your song. You now have a fingerprint for the audio file. Take a look at Conditional Restricted Boltzmann Machines. They are Neural Networks which use multiple binary states to encode time series information. As you can see in the webpage, they trained on human-gait data and can now generate their own human gait. This is essentially what you want but for music files. So you can train CRBMs on the Audio MFCC vectors that you have.

After the training is done, to generate an audio file you can either "seed" the CRBM with a few seconds of some melody or just randomly initialize it. Then just allow the CRBM to go nuts and record whatever it produces. This is your new audio file. To produce another sample use a different seed.

This solves the question of how you can implement a "melody" generation scheme. There are of course variations. You can add other features to you vector apart from MFCC. You can also use other time series predictors like LSTM or Markov models.

All of this being said, the problem of generating music might be much more nuanced than it looks at first glance. Machine Learning algorithms just apply previously learned patterns in the data. How does that correspond to "creating" new music , is a philosophical question. If we analyze the aforementioned algorithm, essentially the CRBM will generate a next output based on the probability distribution that it has learnt. It would be very interesting to see what kind of output it generates when the said distribution is that of musical notes.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.