I'm trying to generate a "beatmap" (from osu!mania) based on an audio file. To do this, I've applied fast fourier transform (this can change to Mel-frequency cepstrum, but that's not super relevant) to the audio, which gets me a 2d array representing the audio. This would be my input.

Input format example: https://i.imgur.com/yWMhwQK.png. It's a 2d array of bytes (0-255) I've generated. Y-axis is time, x-axis is frequency. The value of each pixel represents the power of that frequency at that time.

My output should be a beatmap, which is essentially a list of notes (as shown visualized in this image https://i.imgur.com/MG1LodL.png). I think this could be represented as a list of objects with a column and time (ms) property.

Output format example: https://i.imgur.com/uPaSUub.png, I've parsed an existing beatmap and retrieved the columns an start times for every note.

I don't have much experience with neural networks (as might show from my question), I know the output format may be problematic.

  1. Is a CNN optimal for this? Audio is spacial data so that should work right?

  2. Should this be done in small samples of audio -> small samples of beatmap? Or would a CNN be able to handle the entire audio fft data -> entire beatmap?

I'm planning on taking ~5 second samples of the audio, so the duration of a song won't influence the input sizes.

  1. The output size can differ though, even with a sample of set duration. Is this a problem? Should the output be represented differently? A 2d array where the x-axis represents columns and the y-axis represents time wouldn't work right, as the notes have to be timed perfectly on the beat of the song.

This is just for a hobby project, I'm mainly using this to learn about neural networks. I hope I've explained my questions clearly.

  • $\begingroup$ Checkout "Automatic Music Transcription" for a bunch of relevant research $\endgroup$ – jonnor Apr 6 at 22:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.