Neural Network: how to utilize weakly-/unsupervised data to improve supervised network?

Question

Let's consider one has built a fully-supervised neural network for some task, e.g. localizing an object in various scenes. As you can imagine, it is quite time-consuming to label data: one has to manually localize the object in an image and draw a bounding-box around it - one at a time.

Say we have a normal convolutional neural network (CNN) for the fully-supervised localization, thus something like this:

          2D input image
                |
       convolutional-layer 1
                |
               ...
                |
       convolutional-layer N
                |
      ---flattened output---
                |                    
    fully-connected classifier
                |
           output layer

How can we adapt this architecture to utilize weakly-labelled data, too?

No matter if we have a sliding-window approach, something like OverFeat or probability distributions for actually localizing objects: we always need fully-labelled training data. This is a problem, because labelling data fully-supervised is really time consuming. Hence, fully-labelled data sets are pretty rare.

In contrast to this, typically large amounts of weakly-labelled or unlabeled data exists. In my opinion there is a large potential in weakly-labelled / unlabelled data, due to its vast availability. The problem is using this potential without having to manually label every sample by hand.

That said, my question is: how can weakly-labelled data sets (i.e. "object" vs. "no object") or completely unlabelled data sets be utilized to improve the robustness of fully-supervised architectures, like the mentioned CNN?

Is it generally possible to mix a supervised approach with some unsupervised approach? Like enabling a fully-supervised architecture to somehow utilize weakly-labelled data for training?

Answer 1

You are finding about Semi-supervised object detection algorithm and Weakly-supervised object detection. Semi-supervised object detection uses Supervised-learning term (Your handmade labeled data) and Semi-supervised learning term (Unlabeled data). Weakly-supervised object detection uses coarse-grained data which is imperfect, inaccurate, or partial.

For surveying Weakly-supervised object detection, I recommend this 2 surveys. https://arxiv.org/abs/2104.07918

• Weakly Supervised Object Localization and Detection: A Survey https://arxiv.org/abs/2105.12694
• Deep Learning for Weakly-Supervised Object Detection and Object Localization: A Survey

For Semi-supervised object detection, I don't know surveys direct on it. so I recommend two papers.

• https://arxiv.org/pdf/2107.05031.pdf Semi-Supervised Object Detection with Adaptive Class-Rebalancing Self-Training (CVPR 2021)
• https://proceedings.neurips.cc/paper/2019/file/d0f4dae80c3d0277922f8371d5827292-Paper.pdf Consistency-based Semi-supervised Learning for Object Detection (NeurIPS 2019)

and Most of them uses 'Self-training' scheme in 'Pseudo-label' in 'Semi-supervised learning'. so I recommend a survey on Semi-supervised learning. https://link.springer.com/article/10.1007/s10994-019-05855-6

• A survey on semi-supervised learning ('Machine Learning', 2019)