Export a detailed report with metrics after the training process of a Keras model

Question

I have an image-segmentation model (.h5) which was trained using dice coefficient, recall and precision metrics.

The model was trained using the ImageDataGenerator class with the following code:

history = model.fit_generator(train_data,
                              steps_per_epoch=20,
                              epochs=200,
                              validation_steps=5,
                              validation_data=valid_data,
                              callbacks=callbacks_list)

Thus, I cannot derive, X_test and y_test in order to run functions such as model.evaluate(), model.predict() or classification_report from Sklearn which require X_test and y_test as arguments.

Is it possible to export a report with metrics for all the class and for each class separately using the trained model and without training the model again ?

UPDATE

Based on the answer of @Mohith7548, I'm trying to get x_test and Y_test data in order to run ImageDataGenerator().flow() function. To succeed that, I implemented the following code:

img_path =  "data/balanced_classes/validation/val_imgs"
label_path = "data/balanced_classes/validation/val_labels"
train_data = []
train_label = []
for i in os.listdir(img_path):
    img = io.imread(os.path.join(img_path,i))
    img = img / 255.
    img = trans.resize(img, (256, 256), mode='constant')
    train_data.append(img)

for l in os.listdir(label_path):
    label = io.imread(os.path.join(label_path,l))
    label = label / 255.
    label = trans.resize(label, (256, 256), mode='constant')
    train_label.append(label)

train_data = np.array(train_data)
train_label = np.expand_dims(train_label, axis=-1)

The shapes of X_test is (samples_num, 256, 256, 3) and of y_test is (samples_num, 256, 256, 1)

But when I'm running the following code:

metrics = model.evaluate_generator(test_data, steps=len(X_test) / batch_size)

I get the following error which is raised from the node of my loss function (TverskyLoss):

...
TP = K.sum((inputs * targets))
Node: 'TverskyLoss/mul'
2 root error(s) found.
  (0) INVALID_ARGUMENT:  required broadcastable shapes
     [[{{node TverskyLoss/mul}}]]
     [[assert_greater_equal_1/Assert/AssertGuard/pivot_f/_23/_55]]
  (1) INVALID_ARGUMENT:  required broadcastable shapes
     [[{{node TverskyLoss/mul}}]]
0 successful operations.
0 derived errors ignored. [Op:__inference_test_function_6827]

Here is the code from the TverskyLoss function that I use:

# Tversky loss function
ALPHA = 0.5
BETA = 0.5

def TverskyLoss(targets, inputs, alpha=ALPHA, beta=BETA, smooth=1e-6):
        
        #flatten label and prediction tensors
        inputs = K.flatten(inputs)
        targets = K.flatten(targets)
        
        #True Positives, False Positives & False Negatives
        TP = K.sum((inputs * targets))
        FP = K.sum(((1-targets) * inputs))
        FN = K.sum((targets * (1-inputs)))
       
        Tversky = (TP + smooth) / (TP + alpha*FP + beta*FN + smooth)  
        
        return 1 - Tversky

What am I doing wrong ?

Answer 1

Yes, it is possible to generate evaluation metrics for your image segmentation model without retraining it. One way to do this is to use the model.evaluate_generator() method, which allows you to evaluate the model using a data generator.

You can use the ImageDataGenerator class to generate a data generator for your test data, and pass it to the model.evaluate_generator() method along with the number of steps to use for evaluation. This will return a list of evaluation metrics, such as loss and any metrics that you specified during model compilation (e.g. dice coefficient, recall, precision).

Alternatively, you can also use the model.predict_generator() method to generate predictions for your test data using a data generator, and then use those predictions to calculate evaluation metrics using functions from scikit-learn or other libraries.

Here's an example of how you might use the model.evaluate_generator() method to generate evaluation metrics for your image segmentation model:

# Create a data generator for the test data
test_data = ImageDataGenerator().flow(X_test, y_test, batch_size=batch_size)

# Evaluate the model using the test data generator
metrics = model.evaluate_generator(test_data, steps=len(X_test) / batch_size)

# Print the evaluation metrics
print(metrics)

You can then use these evaluation metrics to generate a report with metrics for all classes and for each class separately, if desired.