Transfer learning VGGish (AudioSet). Impact of zero padding to fit the input size

Question

I am trying to train a network on top of the VGGish architecture (https://github.com/tensorflow/models/tree/master/research/audioset/vggish), using (transfer learning) finetuning.

I initially started out with just finetuning the embedding layers, and training from scratch a simple MLP with some regularization techniques. (2 hidden layers FC with ReLU, BatchNorm and dropout, final output currently is FC to 4 classes). It is able to get an accuracy of 1 on the training data.

The input of the VGGish network are spectograms. My collected audiofiles are pretty low quality OGG/vorbis with a sample rate of 8000Hz. This is lower than the audio used in VGGish, therefore, if I create the log-mel spectogram using their code, every spectogram is missing some bins. (see bands 48 and up in the image below, y-axis) note that this happens in every image, so my hypothesis is that this shouldn't really matter. It is essentially just padding to fit the input size of the VGGish network.

Will this have any negative consequences on the ability of the network to learn using my dataset? Do I also need to finetune the ConvLayers earlier on in the network?

Any other thoughts on how I can work around this (potential) issue?It is not feasible to increase the frequency of the recordings, as they are being streamed from low power devices.

I do not have enough (annotated) data to fully retrain, so training a model from scratch is not feasible.

Suggestions on other pretrained networks are also welcome!

Thanks :)

Answer 1

Will this have any negative consequences on the ability of the network to learn using my dataset?

That's really hard to say, but I suspect it does not make a difference for any sounds that aren't exclusively defined by higher frequencies.

As a rule of thumb: If a human can still correctly identify a sound sampled at 8kHz, so should a machine. If the machine cannot do it, because of the missing higher frequencies, it may not have learned the characteristics of the sound to identify in the first place, but something else.

Do I also need to finetune the ConvLayers earlier on in the network?

I'd start with not re-training those. Having been trained on AudioSet, they should be fairly universal.

Any other thoughts on how I can work around this (potential) issue?It is not feasible to increase the frequency of the recordings, as they are being streamed from low power devices.

Not really. You could upsample your audio to 22.05kHz and go from there—but you cannot create new information for those higher bands out of thin air. So, I don't recommend upsampling.

Suggestions on other pretrained networks are also welcome!

Read Look, Listen, and Learn More: Design Choices for Deep Audio Embeddings by Cramer, Wu, Salamon, and Bello. They offer an open, pre-trained embedding model for environmental sounds at https://github.com/marl/openl3 (which, IIRC, is better than VGGish).

You might be able use its output as input for a shallow dense network to train a classifier for your audio (unless of course you way too few samples).