Unsupervised image segmentation

Question

I am trying to implement an algorithm where given an image with several objects on a plane table, desired is the output of segmentation masks for each object. Unlike in CNN's, the objective here is to detect objects in an unfamiliar environment. What are the best approaches to this problem? Also, are there any implementation examples available online?

Edit: I am sorry, question may have been a bit misleading. What I meant by "unfamiliar environment" is that objects may be unknown to the algorithm. The algorithm shouldn't need to understand what the object is, but should only detect the object. How should I approach this problem?

Answer 1

Fast answear

Mean Shift LSH which is an upgrade in $O(n)$ of the famous Mean Shift algorithm in $O(n^2)$ well know for its image segmentation ability

Some explanations

If you desire a true unsupervised approach to segment images, use clustering algorithms. The fact is that there is a lot of algorithms with different time complexity and specificity. Take the most famous one, the $K$-Means, it is in $O(n)$ so pretty fast but you have to specify how many cluster you want which is not what you intend by exploring an unknown image without any information about how many shapes are presents in it. Moreover even if you suppose that you know how many shape are present, we may suppose that there shapes are random which is another point where the $K$-Means fail because it is design to find elliptic clusters and NOT random shape ones.

At the opposite we have the Mean Shift that is able to find automatically the number of cluster -- which is useful when you don't know what you're looking for -- with random shapes.

Of course you replace the $K$ parameter of $K$-Means by others Mean Shift parameters which can be tricky to fine tuned but it doesn't exist a tool that allow you to do magic if you're not exercising to do magic.

An advice to image segmentation clustering

Transform your color space from RGB to LUV which is better for euclidean distance.

$K$-Means vs Mean Shift LSH time complexity

• Mean Shift : $O(\alpha.n)$
• K-Means : $O(\beta.n)$
• $\alpha \gt \beta$

Mean Shift LSH is slower but it fits better with your needs. It stay still linear and is also scalable with the mentioned implementation.

PS : My profile picture is an application of the Mean Shift LSH on myself if it can help to figure out how it works.

Answer 2

You may need to take a look at this work submitted and accepted for CVPR 2018 : Learning to Segment Every Thing

In this work, they try to segment everything, even objects not known to the network. Mask R-CNN has been used, combined with a transfer learning sub-network, they get very good results in segmenting almost everything.

Answer 3

The state-of-the-art (SOTA) for image segmentation would be Facebook's Mask-RCNN.

While it is usually trained on dataset such like COCO or Pascal which feature real-life objects, you can re-trained it on a dataset of your choice, real or not.

Facebook provides an implementation (Detectron) under the Apache2 license. Give it a try!

Answer 4

Actually, your task is supervised. Segnet can be good architecture for your purpose which one of its implementations can be accessed here. SegNet learns to predict pixel-wise class labels from supervised learning. Therefore we require a dataset of input images with corresponding ground truth labels. Label images must be single channel, with each pixel labelled with its class ....

Also, take a look at Fully Convolutional Networks which are well suited for your task.

---

Based on the edits in the question, I add extra information. There are numerous methods that can be applied for this task. Basically the easiest one is to use a background label and classify those classes that you don't know as background by employing the mentioned architectures. By doing so you will have labels which can have overlap for background class which is a probable downside of this approach but its advantage is that in cases where your trained labels are frequently used in the inputs, you can have a relatively light version of architecture which recognizes the unknown classes.

Answer 5

This might be something that you are looking for. Since you ask for image segmentation and not semantic / instance segmentation, I presume you don't require the labelling for each segment in the image.

The method is called scene-cut which segments an image into class-agnostic regions in an unsupervised fashion. This works very well in case of indoor cluttered environment.

Paper link: arxiv

Code: code