How to represent audio data in a format that can be used for preprocessing and modelling?

Question

I have a project that I am working on currently. The project is to classify audio data. The data is in two folders train and test. In the train folder, there are multiple folders like cat, dog, elephant etc and these folders contain multiple audio files. For example, there are multiple audio files inside the folder dog, cat etc. The folders are also the class names representing the class the audio belongs to. The audio is in .WAV format. How to represent this data in a format on which I can use Machine learning models?

To explain my question a bit: let's take a .csv file. We have rows and columns. Rows are data points and columns are features and training machine training models on this data makes sense.

So, How do I represent this audio data that I can work with and fit ml models?

Answer 1

Audio .wav codec file has a 44 byte header which will give you critical data like bit depth ( CD quality audio is 16 bits per sample), sample rate ( CD quality uses 44,100 audio samples per second ), number of channels, etc ... the balance of bytes in a .wav file is the payload which is the audio curve stored as a set of integers which define the height of the audio curve across a succession of instances in time. The beauty of .wav is its not compressed so all bytes after the header only store the audio curve.

Many audio libraries will allow you read a wav file and return back a floating point array of samples per channel. Visualize a dog bark, this sound propagates from the dog and lands on the membrane of a microphone. Similar wobbling different mediums. This wobbling recorded over time can be represented as a time series curve ... as its digitized its transformed from analog to digital ( ADC ) the original analog audio wave is ~sampled~ X number of times per second to generate a simplification of that analog curve during this analog to digital conversion. Each audio sample represents the height of that audio curve at a specific instant in time.

If you only devote 1 bit to record the audio curve, for a given sample, this will transition from 0 to 1 as the audio curve wobbles above and below the stationary silence flat line mark. Whereas a bit depth of 16 bits which can store 2^16 distinct integers allows far greater granularity and so more accurately models the height of the audio curve.

Unlike image processing where conceptually the entire dataset can be ~consumed~ in an instant of time, audio has an inherent dimension of time. Audio is a time series. Another attribute of audio is the simplicity with which any arbitrary audio curve can be represented by a set of pure sin curves, each with an amplitude, frequency and phase shift. Joseph Fourier outlined the theory whereby time domain data like audio can be represented equally well in either the time domain ( wav codec or analog audio curve ) or in the frequency domain. It does not go unnoticed that this symmetry implies a conservation of information.

To an arbitrary degree of precision a window of audio samples ( in the time domain ) can be transformed using a Fourier Transform into its frequency domain representation with a controllable degree of information loss or lack thereof. Be aware that as you increase the number of samples in this window you decrease the temporal specificity of the resultant spectrogram ... meaning the ability to pin point when a frequency occurred is aided by using the lowest number of audio samples in your window send into the FFT call ... although the frequency resolution in the freq domain is increased by feeding in a larger number of samples to your FFT call ... there is no such thing as a free lunch

So now you have options ... perform your analysis of audio while in its native time domain or send it into a FFT call and have the same information as a set of frequency bins where each bin has values for ( frequency , amplitude, phase shift )

This should give you ideas to tease apart and dig into - take care

Answer 2

Traditionally, spectrograms have been used for most audio ML tasks. Depending on your task, a different kind of spectrogram might be best suited (different frequency resolutions, scaling etc.). In your case, I'd probably try mel magnitude spectrograms that have unit variance and zero mean.

To feed this spectrogram into a deep learning network, like a convolutional neural network (CNN), you'd use fixed length overlapping windows (you do the same at prediction time). Alternatively, other approaches like RNN or most recently TCNs have been used. They all use spectrograms as input. To extract a mel spectrogram (or other formats), you may use a library like librosa and specifically melspectrogram. Examples for audio classification using spectrograms are Genre-Agnostic Key Classification With Convolutional Neural Networks or A Single-Step Approach to Musical Tempo Estimation Using a Convolutional Neural Network. You can find out more about animal sound classification, specifically for birds, at BirdCLEF2019 or at DCASE.

For music, Dieleman showed in End-to-end learning for music audio (IEEE paywalled) that you can also use the waveform directly to classify an audio signal. Later work by Pons et al., End-to-end learning for music audio tagging at scale (not paywalled), explains in more detail whether and why classifying the waveform instead of spectrograms is useful.

I strongly recommend to sticking to spectrograms to get started, simply because IMHO the setup is much easier.