I'm trying to train neural networks to extract parameters out of input data. In particular determining the width, height and color of bricks (see attached pictures as example inputs).
I've already tried CNNs operating on the raw images and NNs operating on extracted features ( 2 describing edges and a color histogram ).
Both approaches seem to learn the mean of the training data. The color is mostly gray and the width and height between the two extremes. The output does not depend on the input as the NN always outputs the "same" values (only very small fluctuations).
As I'm operating on generated data I have a perfect correlation between input and desired output and infinite training data. The used loss function is mean squared error (also tried the absolute error).
Can someone explain why this is happening or how to avoid this training behavior?
I'm using the library tiny-cnn. My current approach (using features) uses 5 fully connected layers, decreasing layer size from 1200 to 17(amount of parameters).
net << fully_connected_layer<sigmoid>(1200, 600)
<< fully_connected_layer<sigmoid>(600, 300)
<< fully_connected_layer<sigmoid>(300, 150)
<< fully_connected_layer<sigmoid>(150, 60)
<< fully_connected_layer<identity>(60, 17);
I've test several CNN structures, always ending in 2 fully connected layers. I tried different kernel sizes from 3 to 11.
This is one of the last I tried:
net << convolutional_layer<tan_h>(100, 100, 5, 3, 4)
<< average_pooling_layer<tan_h>(96, 96, 4, 2)
<< convolutional_layer<tan_h>(48, 48, 7, 4, 8)
<< average_pooling_layer<tan_h>(42, 42, 8, 2)
<< convolutional_layer<tan_h>(21, 21, 7, 8, 16)
<< average_pooling_layer<tan_h>(15, 15, 16, 3)
<< convolutional_layer<tan_h>(5, 5, 5, 16, 32)
<< fully_connected_layer<relu>(32, 64)
<< fully_connected_layer<identity>(64, size);
5-conv, average, 7-conv, average, 7-conv, average(3x3), 5-conv, full, full
I've mainly used RMSProp and adagrad with standard learning rates but also tried higher and lower rates.
Example inputs: