I am using Keras to train different models on the COCO keypoints dataset. All of the models I am working with are used for image segmentation, so they output heatmaps corresponding to the labels.

All of the segmentation models use a binary cross-entropy loss and an accuracy metric. However, I have realized now that these are not fit for keypoint detection because in a 256x256 image, there is only a single '1' label for the keypoint. This drives the loss to 0 and the accuracy to 100% VERY quickly.

What is commonly done for loss/metrics in keypoint detection?


1 Answer 1


You are right for keypoint detection you output heatmaps. However I have come across two types of such heatmaps.

  1. Discrete: This is the type used by KeypointRCNN model in torchvision. If you use this type of target, you can use cross-entropy as loss function. However, the implementation in KeypointRCNN is for a single instance per image region.
  2. Continuos: You use a continuous heatmap as target where each keypoint-type(for example: left shoulder) has one channel. Typically to avoid the problem of many 0s you use guassian kernels to generate the target, where the centre of the 2d guassian kernel is at the keypoint location. In this scenario you use MSE loss function.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.