What is a good method for detection of rare occurencies of speech in noisy audio data?

Question

I have some audio recordings (with relatively static but noisy background, e.g., wind in an open area) with small number of short occurrences of speech (~1% of the total audio duration).

What would be a good method to detect the speech occurences in an unsupervised manner?

I have tried simple thresholding on a spectrogram, but this is problematic since:

• The intensity of the background can wary with time (i.e. noise is louder sometimes)
• Different speech segments need not to be similar to each other
• Often, speech is too quiet (compared to the average loudness of the background) and is overlaid by noise

This may seem like quite a hard task, however I can easily notice the speech segments by listening to the audio/looking at the spectrogram, since spectrogram of speech has some distinct structure (although it is non-trivial to rely on the structure for detection as it is still quite non-regular).

• Note that I just want to detect intervals with something sounding like human speech (or, say, something distinct enough from the background, since data typically has no other sound sources besides background/speech).

Answer 1

Detecting speech specifically is a well known problem, generally called Voice Activity Detection (VAD). The simplest mechanisms just calculate a time-averaged ratio of energy in the speech frequencies compared to total energy, many implementations on Github of this idea. The audio codes for speech also tend to include a VAD component. A modern example can be found in libvad, based on the VAD from WebRTC project.

More generally you can try Anomaly Detection using algorithms like Isolation Forest or LocalOutlierFactor. What is critical is to use good features as input. A good starting point would be to calculate mel-spectrogram or MFCC. The spectrogram frames should be normalized, typically by subtracting the median/mean and dividing by RMS energy. Run the anomaly detection on individual frames of 20-50ms and then do some filtering of classifications to reduce false triggering. A simple method is to require N consecutive frames to be classified as an anomaly to consider it a real anomaly. Or to require at least P percent of frames within a larger rolling window (say 200-1000ms).

In Python, you can use librosa for audio feature extraction and scikit-learn for anomaly detection

Note that for evaluating your method you will need to collect and label a test-set.

Answer 2

If you just need a tool to achieve it, you could use Google's SpeechRecognition package. Here is a tutorial on how to use it.

It's great because it is able to turn on only when it hears some speech and it's amazing even in noisy environments.