3
$\begingroup$

I use spectrogram as input to a Convolutional Neural Network I have created with tensorflow.keras in Python.

Its shape is (time, frequency, 1).

The input's shape of the CNN is (None, time, frequency, n_channels) where n_channels=1 and the first layer is a Conv2D. In between every Convolutional layer I use a BatchNormalization layer before an Activation layer. The default value for BatchNormalization is "axis=-1". Should I leave it as it is or should I make it with "axis=2" which corresponds to the "frequency" axis?

The thought behind this is that the features of a spectrogram are represented in the frequency axis.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Interesting question :)

Using spectrograms means you are essentially using images (of frequencies varying over time). I understand the content is in understood like the graph i.e. with axes time and frequency, but as far as the network knows, you are giving it black and white images; assuming your last dimension (=1) is the channels dimension.

You normally want to take the batch-norm over the features, so it could depend on what you see as a feature. Do you care about the shape of the peaks/troughs of the spectra, or the values encoded in the final channel? The meaning of axis when used in terms of BatchNormalization might be a little confusing. Have a look at these explanations and some of the points here.

So as far as pure images go, I would recommend keeping the default axis=-1.

Remember that a 2d convolution operation is looking for spatial correlations over the image itself - the kernel slides from left to right, top to bottom. So mixing that idea with then taking batch norm over your second axis (frequency) is literally an orthogonal idea.

I don't know if it will work out, but I think it might make most sense if your spectrograms are aligned in time, that is they cover the same nominal time period and so for two of them with the same size, the x-axis has the same scale. Otherwise you would be taking batch-normalization over different time-frames and therefore stirring up temporal information between the samples.

Improve this answer
$\endgroup$
2
  • $\begingroup$ So if i use axis=1 corresponding to the "time" dimension is more reasonable than what I suggested earlier, correct? $\endgroup$
    – E. Vasilopoulos
    Commented Feb 10, 2020 at 15:36
  • 1
    $\begingroup$ It sounds reasonable to normalise over the frequency as it is your feature of interest (that means BatchNormalization takes the mean/variance over time and your "colour" channel. Again, I think time alignment of the samples might be important. However, I honestly am not sure what could make most sense with your data as it will likely depend on what you are trying to predict. I would suggest trying all three possibilities (it is only changing one value and re-running the experiment) and assess the impact. Maybe you will be able to then explain the differences fully to us all :) $\endgroup$
    – n1k31t4
    Commented Feb 10, 2020 at 15:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.