I am attempting to create a 1:N speaker identification model with Keras using a TensorFlow backend. I used the LibriSpeech corpus for training data, and preprocessed the data by first converting each file from .FLAC to .WAV and then calculating the Mel-frequency cepstral coefficients (MFCCs) from the first three seconds of audio. I then fed the MFCCs into a convolutional neural net (CNN) created with the following function:
def createModel(self, model_input, n_outputs, first_session=True): if first_session != True: model = load_model('SI_ideal_model_fixed.hdf5') return model # Define Input Layer inputs = model_input # Define First Conv2D Layer conv = Conv2D(filters=32, kernel_size=(5, 5), activation='relu', padding='same', strides=3, name='conv_1A')(inputs) conv = Conv2D(filters=32, kernel_size=(5, 5), activation='relu', padding='same', strides=3, name='conv1B')(conv) conv = MaxPooling2D(pool_size=(3, 3), padding='same', name='maxpool_1')(conv) conv = Dropout(0.3, name='dropout_1')(conv) # Define Second Conv2D Layer conv = Conv2D(filters=64, kernel_size=(3, 3), activation='relu', padding='same', strides=3, name='conv_2A')(conv) conv = Conv2D(filters=64, kernel_size=(3, 3), activation='relu', padding='same', strides=3, name='conv_2B')(conv) conv = MaxPooling2D(pool_size=(3, 3), padding='same', name='maxpool_2')(conv) conv = Dropout(0.3, name='dropout_2')(conv) # Define Third Conv2D Layer conv = Conv2D(filters=128, kernel_size=(3, 3), activation='relu', padding='same', strides=3, name='conv_3A')(conv) conv = Conv2D(filters=128, kernel_size=(3, 3), activation='relu', padding='same', strides=3, name='conv_3B')(conv) conv = MaxPooling2D(pool_size=(3, 3), padding='same', name='maxpool_3')(conv) conv = Dropout(0.3, name='droupout_3')(conv) # Define Flatten Layer conv = Flatten(name='flatten')(conv) # Define First Dense Layer conv = Dense(256, activation='relu', name='dense_a')(conv) conv = Dropout(0.2, name='dropout_4')(conv) # Define Second Dense Layer conv = Dense(128, activation='relu', name='dense_b')(conv) conv = Dropout(0.2, name='dropout_5')(conv) # Define Output Layer outputs = Dense(n_outputs, activation='softmax', name='output')(conv) # Create Model model = Model(inputs, outputs) model.summary() return model
The model is designed to determine whether the speaker is one of 15 persons of interest, so it was then retrained with transfer learning on the database of 15 speakers. For testing purposes, the database consisted of 14 LibriSpeech speakers that the model had never seen before and myself. During this stage of training, the model achieved a max validation accuracy score of 0.9416:
Epoch 10/100 547/547 [==============================] - 0s 498us/step - loss: 0.1778 - accuracy: 0.9452 - val_loss: 0.2544 - val_accuracy: 0.9416 Epoch 00010: val_accuracy improved from 0.91971 to 0.94161, saving model to SI_ideal_model_fixed.hdf5
Finally, I recorded a live sample of my voice and asked the model to predict whether it was myself or one of the LibriSpeech speakers with the following function:
def predict(self, audio_path): # Import Model model, data = self.importModel(audio_path) # Prepare Audio for Prediction pla = ProcessLiveAudio() file = pla.getMFCCFromRec() # Interpret Prediction prob = model.predict(file) # Make prediction print(prob) index = np.argmax(prob) # Decode one-hot vector prob_max = prob[index] # Answer confidence prediction = data[index] # Determine corresponding speaker # Print Results print('Speaker: ' + prediction) print('Confidence: ' + str(prob_max*100) + ' %')
With the following results (speakers are in the first list, with ZH being myself, and corresponding probabilities are in the second):
['1743', '1992', '2182', '2196', '2277', '2412', '2428', '2803', '2902', '3000', '3081', '3170', '3536', '3576','ZH'] [[1.0116849e-04 9.0038668e-08 9.9856061e-01 5.8844932e-07 2.0543277e-05 5.1232328e-06 3.5524553e-07 5.9479582e-08 7.4726445e-06 6.2108990e-10 2.0075995e-10 2.6086704e-08 1.0949866e-03 2.0887743e-04 7.1733335e-12]]
So, the model not only predicted with 99.86% confidence that I was a completely different person, but also assumed that I was far and away the least likely classifier to which the audio signal belonged. Moreover, the same issue appeared in every subsequent test of the model, sometimes with a different incorrect speaker. Yet, I am confused as to how to improve the model because it obviously excelled during the training stages and avoided significant overfitting. Does the issue stem from training, or is it an issue with my prediction function, or even something entirely different?
TL;DR: What are the best steps to fix a multi-classification Keras model that performs well during training and then confidently predicts the wrong classifier?
I am new to ML/Keras and any help would be greatly appreciated.