Is there a known convolutional net architecture to calculate object masks for images?

Question

I would like to train a convnet to do the following:

1. Input is a set of single channel (from black to tones of grey to white) pictures with a given object, let's say cars.
2. Target is, for every picture in the set, the same picture, however pixels are either black or white. The pixels corresponding to the car object are in white (i.e. intensity 255) and the pixels corresponding to the background are black (i.e. intensity 0).

After training I would like to feed the net with pictures of cars and I would like the prediction - the ideal prediction at least - to be a picture with pixels either black or white, where white corresponds to the object and black to the background.

I assume that the input layer is a 2D convolutional layer and the output layer is also a 2D convolutional layer, each one with as many neurons as pixels in the pictures.

Can anyone please explain what kind of network architecture would accomplish just that?

It could be either the architecture (in theory) or implemented in code.

I expect to tweak it, but it would be nice not to start from scratch.

Answer 1

I'm surprised nobody mention fully convolutional neural networks (FCNs) for semantic segmentation.

They are inspired by the original AlexNet style convnets that ended with one or two densely connected layers and softmax classifier. But FCNs dispense with the dense layers and stay fully convolutional all the way to the end.

Here's the basic principle. Take AlexNet, or VGG or something like that. But instead of using the parameters in the classifier to compute a scalar for each category, use them to compute a whole array (i.e. image) using a 1x1xNUM_CATEGORIES convolution. The output will be NUM_CATEGORIES feature maps, each representing a coarse-grained "heat map" for that category. A map of dogness, a map of catness. It can be sharpened by including information from earlier layers with "skip connections".

EDIT: Just one further bit of good news: the authors of that paper provide implementations of their nets in Caffe's Model Zoo. Tweak away!

Answer 2

this is a typical image segmentation problem. You need to find a continuous blob in the image which is the segment you are looking for. A well-known architecture for this problem is the U-net architecture, and this is the link to the paper: http://arxiv.org/abs/1505.04597

The architecture is called u-net because it has a contracting path (where the size of the input matrix is reduced) and an expanding path (where the size is enlarged again so that the output is more or less similar in size to the input).

The contracting path resembles most closely the typical convnet architecture, and it repeats the following pattern: a) convolutional layers (in this case using a 3x3 kernel), b) a rectified linear unit (RELU) on the convolutional layer output, and c) a pooling layer where the image is halved in every dimension. As you go down the contracting path, the number of feature channels is doubled every time the input goes through a convolutional layer, but the size is reduced every time it goes through a pooling layer.

The expanding path is more or less symmetrical, with upsampling layers that increase (double) the size of the input, followed by a typical convolutional layer and a rectified linear unit (RELU). While you go up the expanding path, the number of feature channels is reduced in the convolutional layers, until you end up with a single matrix (array) that corresponds to the output image.

I look forward to implementing this architecture in the next few days, and I am sure my understanding will increase as a result. If I find out anything additional worth posting, I will go ahead and post it.