# Keras deep learning speaker identification model excels during training and then fails predictions

I am attempting to create a 1:N speaker identification model with Keras using a TensorFlow backend. I used the LibriSpeech corpus for training data, and preprocessed the data by first converting each file from .FLAC to .WAV and then calculating the Mel-frequency cepstral coefficients (MFCCs) from the first three seconds of audio. I then fed the MFCCs into a convolutional neural net (CNN) created with the following function:

def createModel(self, model_input, n_outputs, first_session=True):

if first_session != True:

return model

# Define Input Layer
inputs = model_input

# Define First Conv2D Layer
conv = Conv2D(filters=32,
kernel_size=(5, 5),
activation='relu',
strides=3,
name='conv_1A')(inputs)
conv = Conv2D(filters=32,
kernel_size=(5, 5),
activation='relu',
strides=3,
name='conv1B')(conv)
conv = MaxPooling2D(pool_size=(3, 3), padding='same', name='maxpool_1')(conv)
conv = Dropout(0.3, name='dropout_1')(conv)

# Define Second Conv2D Layer
conv = Conv2D(filters=64,
kernel_size=(3, 3),
activation='relu',
strides=3,
name='conv_2A')(conv)
conv = Conv2D(filters=64,
kernel_size=(3, 3),
activation='relu',
strides=3,
name='conv_2B')(conv)
conv = MaxPooling2D(pool_size=(3, 3), padding='same', name='maxpool_2')(conv)
conv = Dropout(0.3, name='dropout_2')(conv)

# Define Third Conv2D Layer
conv = Conv2D(filters=128,
kernel_size=(3, 3),
activation='relu',
strides=3,
name='conv_3A')(conv)
conv = Conv2D(filters=128,
kernel_size=(3, 3),
activation='relu',
strides=3,
name='conv_3B')(conv)
conv = MaxPooling2D(pool_size=(3, 3), padding='same', name='maxpool_3')(conv)
conv = Dropout(0.3, name='droupout_3')(conv)

# Define Flatten Layer
conv = Flatten(name='flatten')(conv)

# Define First Dense Layer
conv = Dense(256, activation='relu', name='dense_a')(conv)
conv = Dropout(0.2, name='dropout_4')(conv)

# Define Second Dense Layer
conv = Dense(128, activation='relu', name='dense_b')(conv)
conv = Dropout(0.2, name='dropout_5')(conv)

# Define Output Layer
outputs = Dense(n_outputs, activation='softmax', name='output')(conv)

# Create Model
model = Model(inputs, outputs)

model.summary()

return model


The model is designed to determine whether the speaker is one of 15 persons of interest, so it was then retrained with transfer learning on the database of 15 speakers. For testing purposes, the database consisted of 14 LibriSpeech speakers that the model had never seen before and myself. During this stage of training, the model achieved a max validation accuracy score of 0.9416:

Epoch 10/100
547/547 [==============================] - 0s 498us/step - loss: 0.1778 - accuracy: 0.9452 - val_loss: 0.2544 - val_accuracy: 0.9416

Epoch 00010: val_accuracy improved from 0.91971 to 0.94161, saving model to SI_ideal_model_fixed.hdf5


Finally, I recorded a live sample of my voice and asked the model to predict whether it was myself or one of the LibriSpeech speakers with the following function:

def predict(self, audio_path):

# Import Model
model, data = self.importModel(audio_path)

# Prepare Audio for Prediction
pla = ProcessLiveAudio()
file = pla.getMFCCFromRec()

# Interpret Prediction
prob = model.predict(file) # Make prediction
print(prob)
index = np.argmax(prob) # Decode one-hot vector
prob_max = prob[0][index] # Answer confidence
prediction = data[2][index] # Determine corresponding speaker

# Print Results
print('Speaker: ' + prediction)
print('Confidence: ' + str(prob_max*100) + ' %')


With the following results (speakers are in the first list, with ZH being myself, and corresponding probabilities are in the second):

['1743', '1992', '2182', '2196', '2277', '2412', '2428', '2803', '2902', '3000', '3081', '3170', '3536',
'3576','ZH']
[[1.0116849e-04 9.0038668e-08 9.9856061e-01 5.8844932e-07 2.0543277e-05
5.1232328e-06 3.5524553e-07 5.9479582e-08 7.4726445e-06 6.2108990e-10
2.0075995e-10 2.6086704e-08 1.0949866e-03 2.0887743e-04 7.1733335e-12]]


So, the model not only predicted with 99.86% confidence that I was a completely different person, but also assumed that I was far and away the least likely classifier to which the audio signal belonged. Moreover, the same issue appeared in every subsequent test of the model, sometimes with a different incorrect speaker. Yet, I am confused as to how to improve the model because it obviously excelled during the training stages and avoided significant overfitting. Does the issue stem from training, or is it an issue with my prediction function, or even something entirely different?

TL;DR: What are the best steps to fix a multi-classification Keras model that performs well during training and then confidently predicts the wrong classifier?

I am new to ML/Keras and any help would be greatly appreciated.

• Can you elaborate a little bit more on the transfer learning part? Did you used a pre-trained model? You mentioned in the beginning you used the LibriSpeech corpus for training and 'later retrained with transfer learning on the database of 15 speakers'. So you have two datasets? Or are those speakers from the same dataset, but you split them?
– Tinu
Jul 14 '20 at 6:42
• To address your actual question, you like to do speaker identification of N speakers. Therefore your model will learn the features which are connected with each speaker. If you now feed a sample from a new speaker into your model, it will look for features of the N speakers from the training data. I don't think it would be possible to build a model which learns to identify 1:N speakers and generalized to N+1:M speakers without ever train on data from them.
– Tinu
Jul 14 '20 at 6:48
• @Tinu I did not use a pre-trained model (like ResNet50 or VGG16) but rather first trained the model outlined in the first code sample on 15 speakers from LibriSpeech and then reused the weights from that training session to a new training session on a set of 14 unseen speakers (also from LibriSpeech) as well as samples of my own voice. (That's what I meant by transfer learning. I was hoping that some more general ID patterns would carry over to the new data.) There are two different datasets, and the model was trained on both. Sorry for the confusion, and hope this provides clarification.
– Zack
Jul 14 '20 at 14:58
• @Tinu To your second point, do you think it would improve performance to solely train the data on samples of the targeted speakers (i.e. the second dataset that includes my own voice)? The datasets both contain the same amount of speakers (15), but is attempting to initially determine general patterns from a different dataset and then applying those weights during the second training session actually resulting in confusion during predictions?
– Zack
Jul 14 '20 at 15:12
• Now I understand. Your transfer learning approach seems valid to me. How many samples per speaker do you have in the second data set? And how many sample are from you? If there is only a single sample from you, the network might have not enough samples to learn and the penalty for ignoring your samples is low due to the high class imbalance.
– Tinu
Jul 14 '20 at 15:42

Your model is failing to generalize to new data. One possible reason is the new data is not like the training data.

There could be many reasons:

• Different speakers
• Different audio recording process
• Different audio preprocessing
• Different file format conversions
• Different audio post processing
• The deep learning model over-learned the specific training dataset.

One approach is systematically go through every step and make sure the the new data is as close as possible to the training data. If that is not help, then increase regularization in the model.

The other reason for failing to generalize would be that the model is over-fitting and has learned how to identify the individuals in the training data. If including new data into your training set doesn't improve your results, try removing parameters from the model or drop out a layer and see how it affects the test results.