First off, ignore the haters. I started working on ML in Music a long time ago and got several degrees using that work. When I started I was asking people the same kind of questions you are. It is a fascinating field and there is always room for someone new. We all have to start somewhere.
The areas of study you are inquiring about are Music Information Retrieval (Wiki Link) and Computer Music (Wiki Link) . You have made a good choice in narrowing your problem to a single instrument (monophonic music) as polyphonic music increases the difficulty greatly.
You're trying to solve two problems really:
1) Automatic Transcription of Monophonic Music (More Readings) which is the problem of extracting the notes from a single instrument musical piece.
2) Algorithmic Composition (More Readings) which is the problem of generating new music using a corpus of transcribed music.
To answer your questions directly:
I think that this would be an unsupervised learning problem, but I am
not really sure.
Since there are two learning problems here there are two answers. For the Automatic Transcription you will probably want to follow a supervised learning approach, where your classification are the notes you are trying to extract. For the Algorithmic Composition problem it can actually go either way. Some reading in both areas will clear this up a lot.
What features from the sound wave I should extract so that the output music is melodious?
There are a lot of features used commonly in MIR. @abhnj listed MFCC's in his answer but there are a lot more. Feature analysis in MIR takes place in several domains and there are features for each. Some Domains are:
- The Frequency Domain (these are the values we hear played through a speaker)
- The Spectral Domain (This domain is calculated via the Fourier function (Read about the Fast Fourier Transform) and can be transformed using several functions (Magnitude, Power, Log Magnitude, Log Power)
- The Peak Domain (A domain of amplitude and spectral peaks over the spectral domain)
- The Harmonic Domain
One of the first problems you will face is how to segment or "cut up" your music signal so that you can extract features. This is the problem of Segmentation (Some Readings) which is complex in itself. Once you have cut your sound source up you can apply various functions to your segments before extracting features from them. Some of these functions (called window functions) are the: Rectangular, Hamming, Hann, Bartlett, Triangular, Bartlett_hann, Blackman, and Blackman_harris.
Once you have your segments cut from your domain you can then extract features to represent those segments. Some of these will depend on the domain you selected. A few example of features are: Your normal statistical features (Mean, Variance, Skewness, etc.), ZCR, RMS, Spectral Centroid, Spectral Irregularity, Spectral Flatness, Spectral Tonality, Spectral Crest, Spectral Slope, Spectral Rolloff, Spectral Loudness, Spectral Pitch, Harmonic Odd Even Ratio, MFCC's and Bark Scale. There are many more but these are some good basics.
Is it possible, with recurrent neural networks, to output a vector of sequenced musical notes (ABCDEF)?
Yes it is. There have been several works to do this already. (Here are several readings)
Any smart way I can feed in the features of the soundwaves as well as sequence of musical notes?
The standard method is to use the explanation I made above (Domain, Segment, Feature Extract) etc. To save yourself some work I highly recommend starting with a MIR framework such as MARSYAS (Marsyas). They will provide you with all the basics of feature extraction. There are many frameworks so just find one that uses a language you are comfortable in.