Classification on sound data

Question

My goal is to detect a problem in a windturbine. I have a dataset of 2h (1 hour for each class). To keep in mind, it will be embedded on an MCU target, so the neural network have to be less than 10M parameters.

I tried to use Mel Spectrogram then a 2D classification network (MobileNetV3 or EfficientNetV2), I obtained fairly good results about 95% and a loss of 0.15.

The question is : Is it possible to use a 1D classification network ? Without passing with by a 2D classification network.

I read different things :

• https://arxiv.org/pdf/2004.00132.pdf which with a 1D mobileNet obtain an error rate under 3%
• https://towardsdatascience.com/whats-wrong-with-spectrograms-and-cnns-for-audio-processing-311377d7ccd Which on the other hand this article say that it isn't possible.

I have tried, on binary classification, the best result I was obtaining was val loss : 0.7, val accuracy : 0.85, but on an other set it was about 0.5 of accuracy (I don't have a data leak between the two sets). I thought the problem was from the lack of data, so I tried with the dataset nsynth in tensorflow dataset which have about 70Gb on training, on the category instrument family and the result was not good, high loss and low accuracy

What do you think about it ? Can we use 1D CNN for classification ?

Thank you for your answer

Answer 1

It is definitely possible. See for example End-To-End Learning for Music Audio Tagging at Scale by Pons et al. The linked article described classification for more than one class and the music domain, so different. But it explores whether a spectrogram frontend (or mel spectrogram frontend) works better than simply using the waveform, i.e. a 1D frontend. If I remember correctly, the difference depended on the dataset size. In any case, to answer your question: you can use 1D.

But should you?

If you are a domain expert for wind turbines, you probably know what frequency ranges are relevant to detect anomalies. Also, important to know when using spectrograms, do you need to be able to detect repeating patterns? Then the hop size may also be a very relevant parameter. In other words: Check what differentiates your clean data from the "broken" data and tailor your spectrogram's frequency resolution and hopsice (temporal resolution) accordingly. Use your domain knowledge. This may help your network quite a bit and save some parameters.

Another approach to improve your results could be data augmentation. Why not create additional "broken" samples by adding plausible, artificial noise to the clean samples and train on them as well?

Last but not least, you could also choose a completely different route for anomaly detection: auto encoders. See for example this tutorial.

Answer 2

In the case of a mel-spectrogram preprocessing, the Short-Term Fourier Transform (STFT) followed by a mel-filterbank reduction and log-scaling is used. It transforms the 1 dimensional audio waveform into a 2d representation of time-frequency, with a particular time and frequency resolution dependent on the mel/STFT parameters.

It is possible to use a neural network to learn a conceptually similar (but potentially more powerful) transformation. The most straightforward is to create a filterbank with learnable weights. There are several examples of this in the literature, for example:

• MatchboxNet: 1D Time-Channel Separable Convolutional Neural Network Architecture for Speech Commands Recognition
• SincNet

This can be beneficial in an embedded/microcontroller/TinyML setting, because a neural network can be quantized to use 8-bit integer operations which has SIMD available on multiple architectures (ex: ARM Cortex M4F), whereas FFT is not as easily done with reduced precision. This makes it theoretically possible that a learned filterbank can be more computationally efficient. But it will probably require careful tuning of hyperparameters.