# LSTM/RNN model fails on new test data - TFLearn

i'm fairly new to ML and at the moment i'm trying to develop a model that can classify spoken digits (0-9) by extracting mfcc features from audio files.

I trained the model on a data set that consists of 15 speakers and 2400 training examples (240 audio examples for each digit). After 3000 epochs the model has achieved 97% accuracy.

The problem is, when i record my own digit wavs, the model fails to classify them correctly. Why is this happening and what could i do to fix this issue? Is this an example of overfitting?

Extracting mfcc features

  wave, sr = librosa.load(wav_file, mono=True)
mfcc = librosa.feature.mfcc(wave, sr)
mfcc = np.pad(mfcc,((0,0),(0,80-len(mfcc[0]))), mode='constant', constant_values=0)


Parameters

learning_rate = 0.0001
training_iters = 150
batch_size = 64

width = 20  # mfcc features
height = 80  # (max) length of utterance
classes = 10  # digits


Network building

net = tflearn.input_data([None, width, height])
net = tflearn.lstm(net, 128, dropout=0.8)
net = tflearn.fully_connected(net, classes, activation='softmax')
net = tflearn.regression(net, optimizer='adam', learning_rate=learning_rate,
loss='categorical_crossentropy')


Training

model = tflearn.DNN(net, tensorboard_verbose=0)

EPOCHS = 20
for i in range(training_iters):
model.fit(X_train, y_train, n_epoch=EPOCHS, validation_set=(X_val, y_val),
show_metric=True, batch_size=batch_size)


It looks like you are using the same training and test data for each iteration. This is causing your model to be overfit.

You need to use cross-validation. I would recommend using fewer training iterations and doing the following:

1. Split your training data into k chunks where k is your number of training iterations
2. At each training iteration make your training set by combining k-1 chunks and your validation data the remaining chunk. Note that this should be done so that your validation and training sets are never the same across two training iterations. Additionally the same chunk should never be in both your train and validation sets.

You can keep the the accuracy metrics on each training iteration if you want to see how your model's accuracy is converging, but the true accuracy of your model can only be measured on a holdout set which was not used in this validation scheme.

If you have enough data I would recommend doing an 80/20 train/validation initial split on your data, doing steps 1 and 2 on the 80% split and measuring the accuracy on the 20% split.

If you really want to be careful and have a lot of data you could do the following:

1. Perform a 60/20/20 train/validation/test split
2. Split both the train and validation sets by the number of iterations you are using.
3. At each iteration, use k-1 chunks of training data and validate against one chunk from the separate validation set.
• Good catch! I had split my data into train, test, validation before and the 97% accuracy resulted from the test set only which was not part of the training process.. But apparently i used the same train and validation set for every iteration..! – Moras Jun 6 '18 at 16:27
• Didn't work.. I still get false predictions on my own wavs. I reduced the total number of epochs to 1200 btw. – Moras Jun 7 '18 at 15:56
• Is there still a large discrepancy between the measured accuracy on the test split and the final measured accuracy? Do the accuracy numbers from the validation splits converge to the same accuracy measured on the hold out set? – KRyan Jun 7 '18 at 17:40
• The final measured accuracy from the holdout set is almost the same with the one from the validation split (96-97%). So, the model classifies correctly every digit from the entire data set (train & validation & test split). However it still fails with new created digit wavs recorded with my voice or really any other example that's not from the same speakers.. – Moras Jun 7 '18 at 18:04