I want to train an english speech to text model using architecture similar to deepspeech.
In general it has 4 blocks:

  1. feature extraction
    I used melspectrogram. (I used n_mels=80)
    This translates (batch, waveform) -> (batch, time, mel).
    Here I should emphasise that I chose such a parameters of melspectrogram so that I have approximately 100 timestamps for 1 second of audio.
  2. cnn block
    There are 2 options. We can use 2d or 1d convolution.
    I chose to use 1d convolution, because it does not change the time dimension, on the other hand 2d convolution decreases it.
    However, we should have at least 50-70 timestamps for 1 second, because (I examined the Librispeech dataset) people often say more than 20 characters in one second.
    If I had chosen 2d convolution, I would have to make frames with some window_size (time) and hop_size. The number of timestemps would have decreased be a factor of hop_size, but it seemed stupid to set hop_size=1.
    So we have (batch, time, mel) -> (batch, time, channels, features)
    I used usual 1d VGG-like block. I tried 1-2 blocks with (128, 256) channels
  3. rnn block before rnn I reshaped (batch, time, channels, features) -> (batch, time, cnn_features) cnn_features = channels * features
    cnn_features = 3000+-1000 in variations of architectures that I tried. As rnn block I tried 5-layer gru. I tried gru with softmax activation (pytorch) and also tried gru with relu activation, which I found in speechbrain.
    I chose hidden_size=512.
    we have (batch, time, cnn_features) -> (batch, time, rnn_features)
  4. dnn block
    This is just 2-layer dnn with hidden_size=2048 and output_size=27 (the number of tokens for ctc loss)
    hidden layer had softmax activation and output had log_softmax (again for ctc loss) (batch, time, rnn_features) -> (batch, time, num_tokens)

All this blocks I implemented using pytorch.

After this I applied ctc_loss (pytorch implementation).
I had 27 tokens (blank and 26 english letters).
I trained with torch Adam optimizer with learning rate 1e-4

I tried to train for several hours on Librispeech dataset, but the training just got stuck every time after 20-30 minutes giving me something like this after greedy choosing token with max probability:
ttthhhhheeeeeeeeeee (just THE word)
After using dropout extensively I got just random output of letter from the set("theaiys") like this:
I definitely didn't have problems with exploding gradients, maybe I had problems with vanishing gradients. I examind standart deviation of different torch modules. Some rnn modules had std near 1e-7-1e-8. I guess it is pretty small, recalling the fact that learning rate is 1e-4.
My another guess is that my rnn has too many timestamps. I had nearly 80 timestamps for 1 second. Librispeech has examples that are longer than 10 seconds, sos the rnn has input of length more than 800. But I didn't want to lower the timestamp, because I would have got too few timestamps.

Maybe someone has ideas what can be the problem. I do not want to get state of the art performance, I just want to train model from scratch.
I looked for notebooks that train ctc models to understand the parameter of the model, but I could not find such notebooks or githubs. If someone knows where to find one, I would be glad if you share it



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.