I'm trying to generate a "beatmap" (from osu!mania) based on an audio file. To do this, I've applied fast fourier transform (this can change to Mel-frequency cepstrum, but that's not super relevant) to the audio, which gets me a 2d array representing the audio. This would be my input.
Input format example: https://i.imgur.com/yWMhwQK.png. It's a 2d array of bytes (0-255) I've generated. Y-axis is time, x-axis is frequency. The value of each pixel represents the power of that frequency at that time.
My output should be a beatmap, which is essentially a list of notes (as shown visualized in this image https://i.imgur.com/MG1LodL.png). I think this could be represented as a list of objects with a column and time (ms) property.
Output format example: https://i.imgur.com/uPaSUub.png, I've parsed an existing beatmap and retrieved the columns an start times for every note.
I don't have much experience with neural networks (as might show from my question), I know the output format may be problematic.
Is a CNN optimal for this? Audio is spacial data so that should work right?
Should this be done in small samples of audio -> small samples of beatmap? Or would a CNN be able to handle the entire audio fft data -> entire beatmap?
I'm planning on taking ~5 second samples of the audio, so the duration of a song won't influence the input sizes.
- The output size can differ though, even with a sample of set duration. Is this a problem? Should the output be represented differently? A 2d array where the x-axis represents columns and the y-axis represents time wouldn't work right, as the notes have to be timed perfectly on the beat of the song.
This is just for a hobby project, I'm mainly using this to learn about neural networks. I hope I've explained my questions clearly.