Neural Net for regression task on images only learning mean of training data

Question

I'm trying to train neural networks to extract parameters out of input data. In particular determining the width, height and color of bricks (see attached pictures as example inputs).

I've already tried CNNs operating on the raw images and NNs operating on extracted features ( 2 describing edges and a color histogram ).

Both approaches seem to learn the mean of the training data. The color is mostly gray and the width and height between the two extremes. The output does not depend on the input as the NN always outputs the "same" values (only very small fluctuations).

As I'm operating on generated data I have a perfect correlation between input and desired output and infinite training data. The used loss function is mean squared error (also tried the absolute error).

Can someone explain why this is happening or how to avoid this training behavior?

I'm using the library tiny-cnn. My current approach (using features) uses 5 fully connected layers, decreasing layer size from 1200 to 17(amount of parameters).

    net << fully_connected_layer<sigmoid>(1200, 600)
    << fully_connected_layer<sigmoid>(600, 300)
    << fully_connected_layer<sigmoid>(300, 150)
    << fully_connected_layer<sigmoid>(150, 60)
    << fully_connected_layer<identity>(60, 17);

I've test several CNN structures, always ending in 2 fully connected layers. I tried different kernel sizes from 3 to 11.

This is one of the last I tried:

net << convolutional_layer<tan_h>(100, 100, 5, 3, 4)
    << average_pooling_layer<tan_h>(96, 96, 4, 2)
    << convolutional_layer<tan_h>(48, 48, 7, 4, 8)
    << average_pooling_layer<tan_h>(42, 42, 8, 2)
    << convolutional_layer<tan_h>(21, 21, 7, 8, 16)
    << average_pooling_layer<tan_h>(15, 15, 16, 3)
    << convolutional_layer<tan_h>(5, 5, 5, 16, 32)
    << fully_connected_layer<relu>(32, 64)
    << fully_connected_layer<identity>(64, size);

5-conv, average, 7-conv, average, 7-conv, average(3x3), 5-conv, full, full

I've mainly used RMSProp and adagrad with standard learning rates but also tried higher and lower rates.

Example inputs:

Answer 1

I took a deeper look at this problem with some example data (that the OP provided) and some simplifications. I just predicted brick height param as opposed to the 17 example params in the question. I used the Keras library in Python to explore a few different architectures.

Initially I replicated the problem, simple CNNs were predicting identical values for all heights, and the best loss (around 0.0053 on the sample data) was when this predicted the mean very precisely for all inputs. For some reason the network could not find the visual structure that is obviously in the data.

After some experimentation, I had a network that functioned as intended. It was surprisingly deep for such a simple problem. My best guess at the reason why is that the combined overlap of all the convolutions needs to be of the same size order as the features being detected.

Some important details:

• Check that the order of the dimensions of the images when read with the image library you are using match the expected structure that the CNN is working on. If the dimensions are in a different order, this will not cause any error, but it scrambles the interpretation used by the CNN and prevents it learning anything but the mean.

• Avoid using pooling layers, as they obscure pixel locations of features from further layers.

• Check that your parameters are not ambiguous. The two "brick width" parameters seem interchangeable for instance. That makes it much harder for the network to predict them, and it is likely to output a rough mean for the example even if you avoid the problems with architecture.

• The problem is harder to find a working architecture for than I first thought. I am not entirely sure why that is the case, but am guessing it is to do with the size of the features that need to be isolated in order for the network to "measure" them. Eventual working architectures were quite deep e.g. 10 convolutional layers, despite the pixel-level structures being quite obvious and clear in the training data.

Here is a gist of my solution in Python/Keras. Summary of architecture in the gist: 4 times Conv2d layers with 8 feature maps and 5x5 kernel. Dropout 25%. 4 more times Conv2d layers with 16 feature maps and 5x5 kernel. Dropout 50%. Fully connected layer with 256 outputs. Dropout 50%. Fully connected later with 128 outputs. Dropout 50%. Single output neuron. All layers use ReLU activation, except output which is linear.

The example re-sizes images to 50x50 for speed. However, using 100x100 images works fine. I added a couple of convolutional layers to it for that (one each for the 8 features and 16 features section), and get a reasonable loss of ~0.00025 on a validation set. It looks possible to get a lower loss still with more training examples, more epochs and perhaps more feature maps. Many of the predictions were very good, close to the actual value.

Interestingly, the worst errors were for short bricks predicting close to a 2:1 or 3:2 ratio with the ground truth, implying that the network had found correct structure but was being confused by the periodic nature of the bricks. I think this is something that could be resolved with more training examples (and perhaps more features in higher layers) and it should be possible to get much lower loss metric on brick height.

I think that the corresponding tiny-cnn structure looks like this:

net << convolutional_layer<relu>(100, 100, 5, 3, 8)
    << convolutional_layer<relu>(96, 96, 5, 8, 8)
    << convolutional_layer<relu>(92, 92, 5, 8, 8)
    << convolutional_layer<relu>(88, 88, 5, 8, 8)
    << convolutional_layer<relu>(84, 84, 5, 8, 8)
    << dropout(80 * 80 * 8, 0.25)
    << convolutional_layer<relu>(80, 80, 5, 8, 16)
    << convolutional_layer<relu>(76, 76, 5, 16, 16)
    << convolutional_layer<relu>(72, 72, 5, 16, 16)
    << convolutional_layer<relu>(68, 68, 5, 16, 16)
    << convolutional_layer<relu>(64, 64, 5, 16, 16)
    << dropout(60 * 60 * 16, 0.5)
    << fully_connected_layer<relu>(57600, 256)
    << dropout(256, 0.5)
    << fully_connected_layer<relu>(256, 128)
    << dropout(128, 0.5)
    << fully_connected_layer<identity>(128, 1)