2
$\begingroup$

I am attempting to create a 1:N speaker identification model with Keras using a TensorFlow backend. I used the LibriSpeech corpus for training data, and preprocessed the data by first converting each file from .FLAC to .WAV and then calculating the Mel-frequency cepstral coefficients (MFCCs) from the first three seconds of audio. I then fed the MFCCs into a convolutional neural net (CNN) created with the following function:

def createModel(self, model_input, n_outputs, first_session=True):
        
        if first_session != True:
            model = load_model('SI_ideal_model_fixed.hdf5')
            
            return model
        
        # Define Input Layer
        inputs = model_input
    
        # Define First Conv2D Layer
        conv = Conv2D(filters=32,
                      kernel_size=(5, 5),
                      activation='relu',
                      padding='same',
                      strides=3,
                      name='conv_1A')(inputs)
        conv = Conv2D(filters=32,
                      kernel_size=(5, 5),
                      activation='relu',
                      padding='same',
                      strides=3,
                      name='conv1B')(conv)
        conv = MaxPooling2D(pool_size=(3, 3), padding='same', name='maxpool_1')(conv)
        conv = Dropout(0.3, name='dropout_1')(conv)
    
        # Define Second Conv2D Layer
        conv = Conv2D(filters=64,
                      kernel_size=(3, 3),
                      activation='relu',
                      padding='same',
                      strides=3,
                      name='conv_2A')(conv)
        conv = Conv2D(filters=64,
                      kernel_size=(3, 3),
                      activation='relu',
                      padding='same',
                      strides=3,
                      name='conv_2B')(conv)
        conv = MaxPooling2D(pool_size=(3, 3), padding='same', name='maxpool_2')(conv)
        conv = Dropout(0.3, name='dropout_2')(conv)
    
        # Define Third Conv2D Layer
        conv = Conv2D(filters=128,
                      kernel_size=(3, 3),
                      activation='relu',
                      padding='same',
                      strides=3,
                      name='conv_3A')(conv)
        conv = Conv2D(filters=128,
                      kernel_size=(3, 3),
                      activation='relu',
                      padding='same',
                      strides=3,
                      name='conv_3B')(conv)
        conv = MaxPooling2D(pool_size=(3, 3), padding='same', name='maxpool_3')(conv)
        conv = Dropout(0.3, name='droupout_3')(conv)
    
        # Define Flatten Layer
        conv = Flatten(name='flatten')(conv)
    
        # Define First Dense Layer
        conv = Dense(256, activation='relu', name='dense_a')(conv)
        conv = Dropout(0.2, name='dropout_4')(conv)
    
        # Define Second Dense Layer
        conv = Dense(128, activation='relu', name='dense_b')(conv)
        conv = Dropout(0.2, name='dropout_5')(conv)

        # Define Output Layer
        outputs = Dense(n_outputs, activation='softmax', name='output')(conv)
    
        # Create Model
        model = Model(inputs, outputs)
        
        model.summary()
    
        return model

The model is designed to determine whether the speaker is one of 15 persons of interest, so it was then retrained with transfer learning on the database of 15 speakers. For testing purposes, the database consisted of 14 LibriSpeech speakers that the model had never seen before and myself. During this stage of training, the model achieved a max validation accuracy score of 0.9416:

Epoch 10/100
547/547 [==============================] - 0s 498us/step - loss: 0.1778 - accuracy: 0.9452 - val_loss: 0.2544 - val_accuracy: 0.9416

Epoch 00010: val_accuracy improved from 0.91971 to 0.94161, saving model to SI_ideal_model_fixed.hdf5

Finally, I recorded a live sample of my voice and asked the model to predict whether it was myself or one of the LibriSpeech speakers with the following function:

def predict(self, audio_path):
        
        # Import Model
        model, data = self.importModel(audio_path)
        
        # Prepare Audio for Prediction
        pla = ProcessLiveAudio()
        file = pla.getMFCCFromRec()
        
        # Interpret Prediction
        prob = model.predict(file) # Make prediction
        print(prob)
        index = np.argmax(prob) # Decode one-hot vector
        prob_max = prob[0][index] # Answer confidence
        prediction = data[2][index] # Determine corresponding speaker

        # Print Results
        print('Speaker: ' + prediction)
        print('Confidence: ' + str(prob_max*100) + ' %')

With the following results (speakers are in the first list, with ZH being myself, and corresponding probabilities are in the second):

['1743', '1992', '2182', '2196', '2277', '2412', '2428', '2803', '2902', '3000', '3081', '3170', '3536',
'3576','ZH']
[[1.0116849e-04 9.0038668e-08 9.9856061e-01 5.8844932e-07 2.0543277e-05
  5.1232328e-06 3.5524553e-07 5.9479582e-08 7.4726445e-06 6.2108990e-10
  2.0075995e-10 2.6086704e-08 1.0949866e-03 2.0887743e-04 7.1733335e-12]]

So, the model not only predicted with 99.86% confidence that I was a completely different person, but also assumed that I was far and away the least likely classifier to which the audio signal belonged. Moreover, the same issue appeared in every subsequent test of the model, sometimes with a different incorrect speaker. Yet, I am confused as to how to improve the model because it obviously excelled during the training stages and avoided significant overfitting. Does the issue stem from training, or is it an issue with my prediction function, or even something entirely different?

TL;DR: What are the best steps to fix a multi-classification Keras model that performs well during training and then confidently predicts the wrong classifier?

I am new to ML/Keras and any help would be greatly appreciated.

$\endgroup$
  • $\begingroup$ Can you elaborate a little bit more on the transfer learning part? Did you used a pre-trained model? You mentioned in the beginning you used the LibriSpeech corpus for training and 'later retrained with transfer learning on the database of 15 speakers'. So you have two datasets? Or are those speakers from the same dataset, but you split them? $\endgroup$ – Tinu Jul 14 at 6:42
  • $\begingroup$ To address your actual question, you like to do speaker identification of N speakers. Therefore your model will learn the features which are connected with each speaker. If you now feed a sample from a new speaker into your model, it will look for features of the N speakers from the training data. I don't think it would be possible to build a model which learns to identify 1:N speakers and generalized to N+1:M speakers without ever train on data from them. $\endgroup$ – Tinu Jul 14 at 6:48
  • $\begingroup$ @Tinu I did not use a pre-trained model (like ResNet50 or VGG16) but rather first trained the model outlined in the first code sample on 15 speakers from LibriSpeech and then reused the weights from that training session to a new training session on a set of 14 unseen speakers (also from LibriSpeech) as well as samples of my own voice. (That's what I meant by transfer learning. I was hoping that some more general ID patterns would carry over to the new data.) There are two different datasets, and the model was trained on both. Sorry for the confusion, and hope this provides clarification. $\endgroup$ – Zack Jul 14 at 14:58
  • $\begingroup$ @Tinu To your second point, do you think it would improve performance to solely train the data on samples of the targeted speakers (i.e. the second dataset that includes my own voice)? The datasets both contain the same amount of speakers (15), but is attempting to initially determine general patterns from a different dataset and then applying those weights during the second training session actually resulting in confusion during predictions? $\endgroup$ – Zack Jul 14 at 15:12
  • $\begingroup$ Now I understand. Your transfer learning approach seems valid to me. How many samples per speaker do you have in the second data set? And how many sample are from you? If there is only a single sample from you, the network might have not enough samples to learn and the penalty for ignoring your samples is low due to the high class imbalance. $\endgroup$ – Tinu Jul 14 at 15:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.