Current object detection architectures like Faster-RCNN and YOLO seem to be overcomplicated in comparsion with segmentation architectures like Unet. So, why can't we just draw some rectangles around the instances on the segmentation mask (which can be done easily using opencv or skimage)? Such an appro

  • $\begingroup$ Is your question complete? Looks like it got cropped $\endgroup$ Dec 29, 2023 at 11:40

1 Answer 1


At the end of the day, if an architecture satisfies your goal, you should use it. If you can detect objects using a segmentation model, go ahead.

But that's where practice differs from theory. Don't forget that there is a reason that image segmentation and object detection are different tasks. Not every task that can be solved by object detection can be solved by image segmentation, and vice versa. For instance:

  • If you need to classify objects that are not contiguous or particularly rectangular (i.e. you need to segment out the background behind someone's face on a webcam), segmentation will be much more practical
  • If you need to rapidly detect a small object on a picture, you really shouldn't care about the other pixels.
  • Trying to detect objects using segmentation would mean that your training data would need to be at the pixel level, and it might create funny artefacts, especially if your objects are small (your model might just guess that everything is "not your object" because it is what the vast majority of pixels are anyway)

There are also opinionated differences between U-NET and YOLO.

U-Net, specifically, was built with medical imaging in mind. So the architecture should:

  • Learn effectively with smaller amounts of training data
  • Work well with very detailed images
  • Not care too much about speed or performance, as the output quality is paramount

Whereas YOLO was built with this in mind:

  • Speed is of the essence as you'd typically use YOLO with video streams, and it needs to scale well to scenarios where multiple bounding boxes need to be computed
  • Because it's typically used on many images, the exact quality of the bounding boxes is less crucial (as you could maybe average out the bounding boxes of a few frames together or do other sorts of postprocessing)
  • YOLO assumes use cases where training data is widely available

So at the end of the day, the reason object detection architectures might appear "overcomplicated" is because they are optimizing for something different. And the reason U-net appears so simple (I'd rather call it elegant), is because it only has so much data to train with, so if the model was too complex, it wouldn't learn as effectively.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.