1
$\begingroup$

I'm working on a Keras neural net that does key point prediction of body parts (left foot, left knee, left hip, etc.). For each image (X), the target (Y) is a list of coordinates for the keypoints (left_foot_x, left_foot_y, left_knee_x, left_knee_y), etc.). Whenever a keypoint isn't visible the X and Y coordinate equal = 0.

I suspect that leaving the data in this state will lead to bad results because 0 really means NA (i.e. the keypoint isn't visible). Leaving the X and Y coordinates 0 would suggest the keypoint is actually in the corner which is false. What is the right way to structure data to do multi target regressions when the correct output is sometimes NA?

$\endgroup$

1 Answer 1

1
$\begingroup$

One option is to modify your network to output a binary label for each body part, signalling whether the part is visible or not.

You would need the following changes:

  • Modify your network architecture to return a real number in the range $[0, 1]$ for each body part. A sigmoid would be an appropriate activation for the last layer. Its semantics would be the probability that the $i^{th}$ body part is visible.
  • Extend your training and test data with appropriate values for the new labels.
  • Change your loss to only take into account the difference between body part positions (expected vs predicted) if the part was expected to be found (e.g. by multiplying). I'm not very sure whether you should use the real labels or the predicted ones here (or even an average of both).
  • Add a new term (added to the original loss) to your loss with the cross entropy of the body part presence flag. You should probably add a weighting factor $\alpha$ to this new loss term to have a proper scale with respect to the position distance loss term.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.