I'm trying to generate a "beatmap" (from osu!mania) based on an audio file. To do this, I've applied fast fourier transform (this can change to Mel-frequency cepstrum, but that's not super relevant) to the audio, which gets me a 2d array representing the audio. This would be my input.

Input format example: https://i.imgur.com/yWMhwQK.png. It's a 2d array of bytes (0-255) I've generated. Y-axis is time, x-axis is frequency. The value of each pixel represents the power of that frequency at that time.

My output should be a beatmap, which is essentially a list of notes (as shown visualized in this image https://i.imgur.com/MG1LodL.png). I think this could be represented as a list of objects with a column and time (ms) property.

Output format example: https://i.imgur.com/uPaSUub.png, I've parsed an existing beatmap and retrieved the columns an start times for every note.

I don't have much experience with neural networks (as might show from my question), I know the output format may be problematic.

  1. Is a CNN optimal for this? Audio is spacial data so that should work right?

  2. Should this be done in small samples of audio -> small samples of beatmap? Or would a CNN be able to handle the entire audio fft data -> entire beatmap?

I'm planning on taking ~5 second samples of the audio, so the duration of a song won't influence the input sizes.

  1. The output size can differ though, even with a sample of set duration. Is this a problem? Should the output be represented differently? A 2d array where the x-axis represents columns and the y-axis represents time wouldn't work right, as the notes have to be timed perfectly on the beat of the song.

This is just for a hobby project, I'm mainly using this to learn about neural networks. I hope I've explained my questions clearly.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.