How can I detect partially occluded objects using Python?

Question

I'm building a computer vision application using Python (OpenCV, keras-retinanet, tensorflow) which requires detecting an object and then counting how many objects are behind that front object. So, objects are often overlapping.

For example: How many are in this queue?

Here, the aim is to detect the person in the front (foreground) of the queue, and then detect the amount of people behind the front person, despite those behind being occluded.

I have built an object recognition model using keras-retinanet to find the object in the foreground, but now I need to count how many are behind that object. My training data is thousands of images of objects ins straight lines, similar to the image linked above.

However, I have been unable to find a viable way to count the objects behind the front object when they are partially occluded (the rest of the people in the queue).

So far I have tried counting edges (using auto-thresholding), lowering the confidence threshold of my object detection model, and training a new object detection model using only occluded objects as training. These methods have all had very low accuracy.

Do you have any advice or direction on how I can attack this problem?

Answer 1

You could try one of the following techniques:

1. Object Tracking: You can use an object tracking algorithm to track the location of the front object over time, and then count the number of objects that pass by that location. There are many object tracking algorithms available, such as the Kalman filter, Mean-Shift, and CNT trackers, to name a few. You can use OpenCV to implement these algorithms.

2. Density-based Counting: This approach involves estimating the density of objects in the scene and then inferring the count from that density. Density-based counting algorithms use the relationship between the density of objects in an image and the number of objects to estimate the count. This approach can be used when the objects are dense, and the overlap between objects is high. You can use methods such as density-based spatial clustering of applications with noise (DBSCAN) or mean-shift clustering to estimate the density.

3. Semantic Segmentation: You can use semantic segmentation to segment the image into different parts based on the class of the object, and then count the objects in each segment. Semantic segmentation is a type of image segmentation that assigns a class label to each pixel in the image. You can use a pre-trained semantic segmentation model such as Mask R-CNN or DeepLabv3+ to segment the image.

4. 3D Convolutional Neural Network: A 3D CNN can take the temporal dimension into account, allowing it to learn how objects move in a video sequence. This approach can be used to track objects that are partially obscured in the scene and count them as they move.

Some examples of 3D convolutional networks include:

C3D: The C3D (Convolutional 3D) architecture is a popular 3D CNN architecture that was originally designed for action recognition in videos. It consists of eight convolutional layers, five max-pooling layers, and three fully connected layers.

I3D: The I3D (Inflated 3D) architecture is an extension of the popular 2D CNN architecture, Inception, to the 3D domain. It inflates the 2D filters into 3D and applies them to the input video frames. It has achieved state-of-the-art performance on several video-based tasks such as action recognition, video captioning, and video segmentation.

P3D: The P3D (Pseudo 3D) architecture is another 3D CNN architecture that works by dividing the video frames into multiple 2D images and applying 2D CNNs to each of these images. It then combines the features learned from each 2D CNN to produce the final output.

R(2+1)D: The R(2+1)D architecture is a recently proposed 3D CNN architecture that decomposes the 3D convolutions into separate 2D spatial and 1D temporal convolutions. This architecture has shown promising results on video-based action recognition tasks.

Answer 2

You can try CSRNet model. The "official" repo is here.

There are a number of other implementations floating around GitHub.