Input and output feature shapes in CNN for speech recognition

Question

I am currently studying this paper and are trying to understand what exactly the input and output shape is. The paper describes an acoustic model consisting of using cnn-hmm as the acoustic model. The input is a image of mel-log filter energies visualised as spectograms. The paper describes a method for phone recognition in which (as far I understand) applying a CNN on these spectograms with a limited weight sharing scheme should be beneficial for phone recognition.

The input shape as far i understand, is 9-15 frames, which seem a bit confusing, as they don't consider number of phonemes a utterance may have, or the length of them, but simply just "choose" a number of frames to operate with.. The number doesn't seem to be connected with the output in any way - or am I misinterpreting something?

For the output

We used 183 target class labels, i.e., 3 states for each HMM of 61 phones. After decoding, the original 61 phone classes were mapped to a set of 39 classes as in [47] for final scoring. In our experiments, a bigram language model over phones, estimated from the training set, was used in decoding. To prepare the ANN targets, a mono-phone HMM model was trained on the training data set, and it was used to generate state-level labels based on forced alignment.

So the output is divided in to 183 classes, being mapped into HMM's with 3 states for each 61 phonemes, and the ANN target (As I see it target = posterior probability) by training a monophone hmm with forced alignment. I am not sure I understand this process. If the ANN targets are those the CNN should aim/regress to and at the end classify the state based on, Why then process the input?.. why not make a simple DNN that does the regression/classification?

It looks like the improvement lies in the use of forced alignment here, and only on monophone? where is the improvement?

And again how I am supposed to link the input shape and the output shape based this? This would require the audio files to have a certain length, the length of the audio is never specified, so I am assuming that this is not the case.

Answer 1

DNN/CNN prediction(training) is done for 1 frame at a time. The output can be any of the 183 outputs states. Length of the audio files is not a problem since the input to the DNN/CNN is of same dimension only the number of inputs change with audio length.

e.g 1.wav has 500 features and each feature is 39 dimensional and 2.wav has 300 features, the trained model will take a 39 dimensional input and output will be 183 dimensional. So depending on length we'll get different number of outputs.

Since all the frames in an utterances are tested against all the 183 possibilities so the output always remains 183 dimensional. There is no need to specify number of phonemes an utterance can have as everything is being done at frame level.

Frame concatenation (9-15 frames) is done to leverage contextual properties of speech data. Phone changes are context dependent. For 15 frame context, we change the input of DNN to [7*39 (left_context) 39 7*39(right_context)], a 585 dimensional vector. So now DNN will take 585 dimensional data as input and will output a 183 dimensional vector.

CNN input

There exist several different alternatives to organizing these MFSC features into maps for the CNN. First, as shown in Fig. 1(b), they can be arranged as three 2-D feature maps, each of which represents MFSC f eatures (static, delta and delta-delta) distributed along both frequency (using the fre- quency band index) and time (using the frame number within each context window). In this case, a two-dimensional con- volution is performed (explained below) to normalize both frequency and temporal variations simultaneously. Alterna- tively, we may only consider normalizing frequency variations. In this case, the same MFSC features are organized as a number of one-dimensional (1-D) feature maps (along the frequency band index), as shown in Fig. 1(c). For example, if the context window contains 15 frames and 40 fi lter banks are used for each frame, we will construct 45 (i.e., 15 times 3) 1-D feature maps, with each map having 40 dimensions, as shown in Fig. 1(c). As a result, a one-dimensional convolution will be applied along the frequency axis. In this paper, we will only focus on this latter arrangement found in Fig. 1(c), a one-dimensional convolution along frequency

So the input to the CNN will be an image patch of size 45 * 40 regardless of the length of the audio file, just the number of such inputs will depend on the length of audio file.

Why force alignment?

Now we are doing everything at frame level so for each frame we need the state labels. Usually this timing information is not available.

Transcribed data usually looks like this, 1.wav -> I am a cat

Now I don't know how many frames belong to I or to a.

HMMs are trained on this data and force alignment is done to generate state level labels for each frame.

Better modeling of input-output relation gives improvement over other methods. Filter bank features also contribute to improvements. A DNN trained with filter bank features gives better performance as compared a DNN trained with MFCC features.