Training a CNN with limited weight sharing

Question

I am currently working with speech recognition, in which i would like to try to use CNN instead of the normal feature extraction step.

I been reading this paper which proposes method using cnn. The input is a visual representation of mel-log filter bank energies of audio files.

And the output is phoneme recognised for each third frame section a portion (a frames, b frequency_bands) of the image.

The network is a CNN, and they propose a different weight sharing - limited weight sharing, since the patterns being seeked for doesn't occur equally anywhere on the image, but localised to certain frequency areas.

Using separate sets of weights for different frequency bands may be more suitable since it allows for detection of distinct feature patterns in different filter bands along the frequency axis. Fig. 5 shows an example of the limited weight sharing (LWS) scheme for CNNs, where only the convolution units that are attached to the same pooling unit share the same convolution weights. These convolution units need to share their weights so that they compute comparable features, which may then be pooled together.

I am not sure I understand the concept of this weight sharing..

Should the weights be shared for each frame but limited in frequency range?

or should it both be limited for each frame and frequency range?

They made a illustration of this weight sharing:

From what i can decipher from the image - is Limited weight sharing option 2.

Each frame do not have the same weight, multiple convolution is applied on to the same frame, and the convolution for the next frame starts at a lower frequency than than the previous frame and has stride = 2. So somehow is the convolution only performed in the diagonal of the image... sounds weird?

Sounds like i've misinterpreted something incorrectly here? Any ideas on how to implement?

Answer 1

It's important to note that before doing this, they split the frames into individual feature maps. Note Fig. 1:

The "45 rows" in the upper left of your diagram (Fig 4.) corresponds to 15 frames times 3 feature maps.

The convolution is a 1D convolution, so they are not actually convolving across multiple frames, only across frequencies. The authors then sum across feature maps in the normal way, meaning summing across frames and features.

Another way of looking at it is in Eq. 9

The variable $O_i$ is a vector across frequency bands only. The sum is over feature maps which includes frames.