I am currently studying this paper and are trying to understand what exactly the input and output shape is. The paper describes an acoustic model consisting of using cnn-hmm as the acoustic model. The input is a image of mel-log filter energies visualised as spectograms. The paper describes a method for phone recognition in which (as far I understand) applying a CNN on these spectograms with a limited weight sharing scheme should be beneficial for phone recognition.

The input shape as far i understand, is 9-15 frames, which seem a bit confusing, as they don't consider number of phonemes a utterance may have, or the length of them, but simply just "choose" a number of frames to operate with.. The number doesn't seem to be connected with the output in any way - or am I misinterpreting something?

For the output

We used 183 target class labels, i.e., 3 states for each HMM of 61 phones. After decoding, the original 61 phone classes were mapped to a set of 39 classes as in [47] for final scoring. In our experiments, a bigram language model over phones, estimated from the training set, was used in decoding. To prepare the ANN targets, a mono-phone HMM model was trained on the training data set, and it was used to generate state-level labels based on forced alignment.

So the output is divided in to 183 classes, being mapped into HMM's with 3 states for each 61 phonemes, and the ANN target (As I see it target = posterior probability) by training a monophone hmm with forced alignment. I am not sure I understand this process. If the ANN targets are those the CNN should aim/regress to and at the end classify the state based on, Why then process the input?.. why not make a simple DNN that does the regression/classification?

It looks like the improvement lies in the use of forced alignment here, and only on monophone? where is the improvement?

And again how I am supposed to link the input shape and the output shape based this? This would require the audio files to have a certain length, the length of the audio is never specified, so I am assuming that this is not the case.


1 Answer 1


DNN/CNN prediction(training) is done for 1 frame at a time. The output can be any of the 183 outputs states. Length of the audio files is not a problem since the input to the DNN/CNN is of same dimension only the number of inputs change with audio length.

e.g 1.wav has 500 features and each feature is 39 dimensional and 2.wav has 300 features, the trained model will take a 39 dimensional input and output will be 183 dimensional. So depending on length we'll get different number of outputs.

Since all the frames in an utterances are tested against all the 183 possibilities so the output always remains 183 dimensional. There is no need to specify number of phonemes an utterance can have as everything is being done at frame level.

Frame concatenation (9-15 frames) is done to leverage contextual properties of speech data. Phone changes are context dependent. For 15 frame context, we change the input of DNN to [7*39 (left_context) 39 7*39(right_context)], a 585 dimensional vector. So now DNN will take 585 dimensional data as input and will output a 183 dimensional vector.

CNN input

There exist several different alternatives to organizing these MFSC features into maps for the CNN. First, as shown in Fig. 1(b), they can be arranged as three 2-D feature maps, each of which represents MFSC f eatures (static, delta and delta-delta) distributed along both frequency (using the fre- quency band index) and time (using the frame number within each context window). In this case, a two-dimensional con- volution is performed (explained below) to normalize both frequency and temporal variations simultaneously. Alterna- tively, we may only consider normalizing frequency variations. In this case, the same MFSC features are organized as a number of one-dimensional (1-D) feature maps (along the frequency band index), as shown in Fig. 1(c). For example, if the context window contains 15 frames and 40 fi lter banks are used for each frame, we will construct 45 (i.e., 15 times 3) 1-D feature maps, with each map having 40 dimensions, as shown in Fig. 1(c). As a result, a one-dimensional convolution will be applied along the frequency axis. In this paper, we will only focus on this latter arrangement found in Fig. 1(c), a one-dimensional convolution along frequency

So the input to the CNN will be an image patch of size 45 * 40 regardless of the length of the audio file, just the number of such inputs will depend on the length of audio file.

Why force alignment?

Now we are doing everything at frame level so for each frame we need the state labels. Usually this timing information is not available.

Transcribed data usually looks like this, 1.wav -> I am a cat

Now I don't know how many frames belong to I or to a.

HMMs are trained on this data and force alignment is done to generate state level labels for each frame.

Better modeling of input-output relation gives improvement over other methods. Filter bank features also contribute to improvements. A DNN trained with filter bank features gives better performance as compared a DNN trained with MFCC features.

  • $\begingroup$ thanks for the response.. So basically i have to resize all my matrices to fit the size (45*40).. and my utterances has to be at a frame level vs having start and end timestamps for both each words and phonemes appearing in each audio file. $\endgroup$ Mar 29, 2017 at 5:56
  • $\begingroup$ You are not resizing. Frame level features are used to have a fixed dimensional learning model. Even if you have time stamps at word/phoneme level then state level labels are still unknown. $\endgroup$ Mar 29, 2017 at 6:00
  • $\begingroup$ If not resizing then what? So you are saying that having the time stamps might not be that helpfull.. But I would need utterances at a frame level instead of the time stamped ones. $\endgroup$ Mar 29, 2017 at 8:39
  • $\begingroup$ It's kinda resizing (Sorry!). Having time stamps is useful when utterances are long (we can cut and make shorter utterances) since viterbi decoding doesn't work well for longer utterances. For a dataset like TIMIT (used in the paper) where utterances are short( < 15 sec) having/not having time stamps for phonemes won't make difference in ASR. $\endgroup$ Mar 29, 2017 at 8:51
  • $\begingroup$ I guess i have to do some resizing to ensure that the the images is separable with the image patches shape. My utterance are short (absolutely < 15 sec).. so having the time stamps for either the phonemes or the word don't make sense?... $\endgroup$ Mar 29, 2017 at 9:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.