Detecting voice in a noisy environment

Question

I'm trying to detect voice in a noisy environment. This is a New Zealand suburban/bush soundscape. We're using low quality microphones (but I don't think this has an effect on my problem) to periodically record the soundscape in an effort to record birdsong. One of the main challenges is we need to remove all potential human voice for privacy reasons. I'm stuck on the problem of how to detect voice in a noisy environment.

Here's an example of a spectogram with a person speaking 8 steps away from the microphone. There is crackle in the recording (don't believe this has an effect) and a bird chirping (activity around 2000hz) in the background. You can see the voice at 256-1024hz from 8sec to 13sec. The activity around 128hz from 15 to 22sec is a vehicle.

I'm trying to find a way of recognising voice activity (with high degrees of precision) just like in this file.

I believe that using a Machine Learning algorithm will have the best success due to the variability of voice and these sound files. (I have also been working on a mathmatical method based on energy proportion in the voice freq ranges per sample but this just doesn't seem to work with the variability of the sound files).

We only have around 10minutes of voice recorded like in the above spectogram. We then probably have about 24hours of non-voice recordings.

Q1: Will this be enough data, if not, would using data from freesound.org be advisable?

Q2: What types of algorithms would work best? - processing time is not a worry, the data contains noise from a multitude of sources (vehicles, animals, people, weather), I don't have much voice to work with(but I expect I can just use voice files from freesound.org) and I want to achieve 100% precision with not so much of a worry about recall.

Q3: Does anyone know of any projects in a similar space to mine where I could see what they've done? (there has been work done on voice detection [VAD] but from a few experiments it all isn't suited to as noisy environments as mine ie. this)

Q4: Last question, does anyone know any efficient techniques to label discrete events in continuous audio data to use in ML algorithms in Python? (At the moment I expect I'll be listening to files on Audacity and recording where voice starts and stops via a spreadsheet)

Otherwise, any other advice would be awesome!

(FYI I have done courses looking at ML-regression, Neural Nets and SVM/tree's but wouldn't call my self 'ML proficient')

Also, are there any other places I can ask these questions? I realise DSP.stack is more suited to solving small problems in the dsp area but I don't know where else to ask this!

Answer 1

This is in fact a very well suited problem for machine learning and has been explored in the past as published here. It is often called the cocktail party problem, because of the way our human brains can focus on a very particular voice pattern despite being drowned by countless other voices all around us.

I can think of two ways by which this problem can be approached the first is to transform the spectrogram to only keep the birds chirping. The second, would be to extract the voice patterns from a sample and then substract the extracted voices from the original sample. This second approach would not remove the noise from the truck.

I would go with the former technique. I would get a few thousands clips of birds chirping. Then introduce some other sounds into it. The mixed sample would be my input $X$ and the target $Y$ would be the clean bird sample. This was you can simulate many instances. However, you will need enough bird chirping data to cover the wide distribution which exists for birds making noise.

Q1

For this to work well, you will need A LOT of data. Yes, you should definitely try to get as many samples from wherever you can. You can then randomly introduce these human voices with the bird clips in order to get the dirty signals, then use these to train your model.

Q2

You most likely will not achieve 100% accuracy with existing machine learning methods. However, if you get close to 90% that is already a miracle and will truly be great. If you are using spectrograms I would first try convolutional neural network (CNN). These have been shown to produce very good results here. The same paper then proposes an autoencoder to clean up the spectrograms. I think this has a high probability of working, but will introduce smoothing to the spectrogram which may distort the birds chirping. Lastly, I would suggest using a generative adversarial network (GAN).

Q3

I would look into the cocktail party problem. You can also get some insight as to what techniques to use by looking at the extensive work that has been done on spectrograms in MRI spectroscopy (MRSI).

Q4

Like I said above I would suggest going from a $n \times n$ sample containing a mixture of sounds to a clean $n \times n$ sample with only the bird chirping. I think this strategy is the one where you will find the most existing literature and many groups have shown it to work.