1
$\begingroup$

I am trying to train a network on top of the VGGish architecture (https://github.com/tensorflow/models/tree/master/research/audioset/vggish), using (transfer learning) finetuning.

I initially started out with just finetuning the embedding layers, and training from scratch a simple MLP with some regularization techniques. (2 hidden layers FC with ReLU, BatchNorm and dropout, final output currently is FC to 4 classes). It is able to get an accuracy of 1 on the training data.

The input of the VGGish network are spectograms. My collected audiofiles are pretty low quality OGG/vorbis with a sample rate of 8000Hz. This is lower than the audio used in VGGish, therefore, if I create the log-mel spectogram using their code, every spectogram is missing some bins. (see bands 48 and up in the image below, y-axis) note that this happens in every image, so my hypothesis is that this shouldn't really matter. It is essentially just padding to fit the input size of the VGGish network.

Will this have any negative consequences on the ability of the network to learn using my dataset? Do I also need to finetune the ConvLayers earlier on in the network?

In this image a mel-spectogram of one of my samples is visible. Note that bands 48 and up are empty. (y-axis)

Any other thoughts on how I can work around this (potential) issue?It is not feasible to increase the frequency of the recordings, as they are being streamed from low power devices.

I do not have enough (annotated) data to fully retrain, so training a model from scratch is not feasible.

Suggestions on other pretrained networks are also welcome!

Thanks :)

$\endgroup$
2
  • $\begingroup$ What is the task you wan to transfer-learn for? I.e. what kind of audio do you want to classify? $\endgroup$
    – hendrik
    Jul 11 '19 at 13:34
  • $\begingroup$ They are also environmental sounds. Currently only 3 different sounds, as proof of concept. They are: Airplanes, Birds, Trains. $\endgroup$
    – Anton S
    Jul 11 '19 at 14:05
0
$\begingroup$

Will this have any negative consequences on the ability of the network to learn using my dataset?

That's really hard to say, but I suspect it does not make a difference for any sounds that aren't exclusively defined by higher frequencies.

As a rule of thumb: If a human can still correctly identify a sound sampled at 8kHz, so should a machine. If the machine cannot do it, because of the missing higher frequencies, it may not have learned the characteristics of the sound to identify in the first place, but something else.

Do I also need to finetune the ConvLayers earlier on in the network?

I'd start with not re-training those. Having been trained on AudioSet, they should be fairly universal.

Any other thoughts on how I can work around this (potential) issue?It is not feasible to increase the frequency of the recordings, as they are being streamed from low power devices.

Not really. You could upsample your audio to 22.05kHz and go from there—but you cannot create new information for those higher bands out of thin air. So, I don't recommend upsampling.

Suggestions on other pretrained networks are also welcome!

Read Look, Listen, and Learn More: Design Choices for Deep Audio Embeddings by Cramer, Wu, Salamon, and Bello. They offer an open, pre-trained embedding model for environmental sounds at https://github.com/marl/openl3 (which, IIRC, is better than VGGish).

You might be able use its output as input for a shallow dense network to train a classifier for your audio (unless of course you way too few samples).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.