How to detect if a person is interacting (as in touching) some object in an image?

Question

I am currently working on a human interaction problem, which tries to identify if a person touched a predefined object.

Current Approach: I am using open-pose to estimate the pose of the person then manipulating the keypoints returned into some logic (simple if else statements) which tells whether person interacted or not. However it seems to be a dirty workaround to me and is not a full proof solution. So is there any better approach I can go for?

Is there any predefined model or white paper I can refer for this problem?

I am struggling on how to detect the pose of a person in an image? Is there a way to classify the pose of a person in three categories like front, back and side?

Answer 1

I think you can use multiple object detection algorithm one for human and one for the object. And then check both positions to see if bounding lines are crossing each other.

Answer 2

There are papers that show how to estimate different poses. For example this one gives an overview of different methods: PoseTrack: A Benchmark for Human Pose Estimation and Tracking. You can take this benchmark and group all the poses into needed categories. For example, by manually assigning your groups. And then you use the categorization of poses like in paper and convert to the poses you need.

One of the solutions for pose tracking is Mask R-CNN. It uses a neural network with convolutional layers and predicts in parallel class, box and binary mask. Here is the original Mask R-CNN paper. And here is the ProTracker algorithm that achieved one of the best results on PoseTrack.