Tensorflow speech-to-text training on single file with tf.nn.ctc_loss not converging

Question

I want to train speech-to-text model on tensorflow.

Code at: https://github.com/bikramjitroy/speech-to-text/blob/master/TrainingUnitTest-CTCTrain.py

A sample "audio.wav" and "label.txt" file also present in repo. https://github.com/bikramjitroy/speech-to-text

I am loading a wav file. Getting 20 mfcc feature per frame. Passing input to one layer RNN (256 hidden unit) and then added a fully connected layer and then using ctc_loss.

The accuracy is not increasing even with 500 steps with one training example :

Error = 70.932137 # Expecting it should go near zero

Accuracy = 0.62068963 # expecting this will go to zero

I have tried bi-directional rnn as well but that is diverging.

What am I doing wrong in code?

Answer 1

I have found the solution.

Earlier I have a Dropout layer after single layer of RNN. I have changed the rnn_layer function as below.

• Removed the DropoutWrapper from the RNN layer - This helped me to overfit
• Added 2 layers of LSTM cell. Single layer LSTM cell taking time to converge but rather than making LSTM layer wide changed it to deep. This helped me achieve faster convergence than making network wider.

Below is the code of rnn_layer.

def rnn_layer(input_tensor, n_cell_units, dropout, seq_length, batch_size):
    lstm_fw_cell = tf.contrib.rnn.BasicLSTMCell(2*n_cell_units, forget_bias=1.0, state_is_tuple=True)
# Dropout layer dropped
#    lstm_fw_cell = tf.contrib.rnn.DropoutWrapper(lstm_fw_cell, input_keep_prob=dropout, output_keep_prob=dropout)
# Multi layer RNN
    lstm_fw_cell = tf.contrib.rnn.MultiRNNCell([lstm_fw_cell] * 2, state_is_tuple=True)

    outputs, output_states = tf.nn.dynamic_rnn(cell=lstm_fw_cell,
                                               inputs=input_tensor,
                                               dtype=tf.float32,
                                               time_major=True,
                                               sequence_length=seq_length)

    return outputs