19
$\begingroup$

In the Keras blog on training convnets from scratch, the code shows only the network running on training and validation data. What about test data? Is the validation data the same as test data (I think not). If there was a separate test folder on similar lines as the train and validation folders, how do we get a confusion matrix for the test data. I know that we have to use scikit learn or some other package to do this, but how do I get something along the lines of class wise probabilities for test data? I am hoping to use this for the confusion matrix.

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ You can call the model.predict_generator(...) function with a generator that reads data from a directory containing the test set. It returns the predictions, which you can use to calculate a confusion matrix. Is that what you're looking for? See here for docs: keras.io/models/sequential $\endgroup$ – stmax Sep 7 '16 at 15:30
  • 1
    $\begingroup$ Yes, I did see that. predict_generator returns a list of predictions which is a list of float values between 0 and 1. How do I interpret this? It cannot be directly used with the confusion matrix. $\endgroup$ – Raghuram Sep 7 '16 at 15:34
  • 2
    $\begingroup$ I haven't tried predict_generator yet (it's rather new), but it seems to return class probabilities. Try to convert values <= 0.5 to 0 and > 0.5 to 1. Once you have a list consisting of 0s and 1s you can feed it to the function for calculating the confusion matrix. $\endgroup$ – stmax Sep 7 '16 at 18:44
  • 2
    $\begingroup$ As an aside, this will work fine for two class problems, but what if there are more than two classes? $\endgroup$ – Raghuram Sep 8 '16 at 0:32
  • 1
    $\begingroup$ If there are more than two classes, your network needs more than one output. For n classes you have n outputs and you predict the class that has the highest output. Have a look at the softmax function (en.wikipedia.org/wiki/Softmax_function). $\endgroup$ – stmax Sep 8 '16 at 7:46
17
$\begingroup$

To get a confusion matrix from the test data you should go througt two steps:

  1. Make predictions for the test data

For example, use model.predict_generator to predict the first 2000 probabilities from the test generator. 

generator = datagen.flow_from_directory(
        'data/test',
        target_size=(150, 150),
        batch_size=16,
        class_mode=None,  # only data, no labels
        shuffle=False)  # keep data in same order as labels

probabilities = model.predict_generator(generator, 2000)
  1. Compute the confusion matrix based on the label predictions

For example, compare the probabilities with the case that there are 1000 cats and 1000 dogs respectively. 

from sklearn.metrics import confusion_matrix

y_true = np.array([0] * 1000 + [1] * 1000)
y_pred = probabilities > 0.5

confusion_matrix(y_true, y_pred)

Additional note on test and validation data

The Keras documentation uses three different sets of data: training data, validation data and test data. Training data is used to optimize the model parameters. The validation data is used to make choices about the meta-parameters, e.g. the number of epochs. After optimizing a model with optimal meta-parameters the test data is used to get a fair estimate of the model performance.

Improve this answer
$\endgroup$
3
  • 3
    $\begingroup$ Thanks for the code snippets. Could you link those two? In your example y_true seems to be populated with dummy data. Would you use generator.classes to populate the array? $\endgroup$ – Gegenwind Mar 10 '18 at 14:10
  • $\begingroup$ I'm not certain, but I think instead of np.array([0] * 1000 + [1] * 1000) you can get the same array by doing generator.classes $\endgroup$ – Mehdi Nellen Jan 2 '19 at 13:28
  • $\begingroup$ Does model.predict_generator automatically run test phase (ie, deativate dropout)? $\endgroup$ – Black Jack 21 Apr 24 '20 at 9:22
2
$\begingroup$

Here is some code I tried and worked for me:

pred= model.predict_generator(validation_generator, nb_validation_samples // batch_size)
predicted_class_indices=np.argmax(pred,axis=1)
labels=(validation_generator.class_indices)
labels2=dict((v,k) for k,v in labels.items())
predictions=[labels2[k] for k in predicted_class_indices]
print(predicted_class_indices)
print(labels)
print(predictions)

You can then use:

print (confusion matrix(predicted_class_indices,labels)

Make sure you use shuffle=False in your test generator (in my case it's validation generator) and reset it using validation_generator.reset() before you make your predictions.

Improve this answer
$\endgroup$
0
$\begingroup$

For confusion matrix you have to use sklearn package. I don't think Keras can provide a confusion matrix. For predicting values on the test set, simply call the model.predict() method to generate predictions for the test set. The type of output values depends on your model type i.e. either discrete or probabilities.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for the answer. I do know that Keras doesn't have its own confusion matrix package. My question is that model.predict_generator returns a list of float values which cannot be used to compute the confusion matrix. $\endgroup$ – Raghuram Sep 7 '16 at 17:00
  • $\begingroup$ What kind of data are you experimenting on? $\endgroup$ – enterML Sep 7 '16 at 18:28
  • $\begingroup$ I am working on images. $\endgroup$ – Raghuram Sep 8 '16 at 0:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.