I’ve trained my image segmentation models using SegmentationModelsPytorch. Three annotators marked up objects. All pixels that were voted as an object pixel by two annotators were marked as the pixels of the object. Each picture was divided in crops, enriched with crops with target pixels (reflection and rotations by 90/180/270 degrees).

Key lines of code:

loss = smp.utils.losses.BCELoss() + smp.utils.losses.DiceLoss()
metrics = [smp.utils.metrics.IoU(threshold=torch.tensor(0.3))]


for i in range(0, EPOCHS):
    train_logs = train_epoch.run(train_loader)
    valid_logs = valid_epoch.run(valid_loader)
    is_best = max_score < valid_logs['iou_score']
    model_name = ''
    if max_score < valid_logs['iou_score']:
        max_score = valid_logs['iou_score']
        model_name = f'{dt}_{i}_score_{round(max_score, 5)}.pth'
        model_full_path = path.join('models', model_name)
        torch.save(model, model_full_path)

Somehow I get Intersection over Union (IoU) of 0.19, 0.28, 0.20 for pairs of dermatologists. Model reports IoU = 0.87 in the best case in training. When measuring IoU of the prediction with the ground truth on the same test set I get IoU = 0.81

So, how is it possible that averaging yields such an improvement (from 0.19-0.28) to 0.87? Why does the actual measured IoU on the same test set differs from the IoU, reported by the model during training (0.87 vs 0.81)?

You can see the training code https://pastebin.com/Vf2By09U And the measurement code https://pastebin.com/wtD4haej



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.