I am a bit unsure about optimizing a neural net with 3 or more layers. The input data is quite noisy and I seem to project the noise into the learning (strong bias in the data, 90% belong to one class out of five).

However, I would like to get some feedback on the interpretation (I use 50/25/8/8/8neurons with dropout (keep_prob=0,9 after first hidden layer):

enter image description here enter image description here

  • $\begingroup$ As far as of today, I figured out that a lower learning rate drastically reduces the variability on the output KPIs. I.e. noise is reduced by learning slower which I interpret as less deviations in the Stochastic Gradient / ADAM. $\endgroup$ – Frank Apr 25 '16 at 17:04

The overal logarithmic shape of your f1 score graph indicates that learning is effective and cost is heading towards some minimum. That's good. I'm assuming that the noise you're referring to is the instability of the graph after approximately 3k iterations: cost dropping and rising in a zig-zag manner.

This often hints at the learning rate being too large. Back propagation finds the right gradient but you take too big step and end up climbing rather than descending along the edge of the cost function. It's especially evident when a graph seems to oscillate around some middle value. You haven't mentioned what exact value of learning rate you're using but try to reduce it. A good starting point would be 0.01 but it depends on many factors so try to experiment.

Another issue might be a batch size: that is, how many examples contribute to the calculation of a gradient. If it's too large you might end up with an average gradient pointing in the wrong direction. And then even a small step (i.e., low learning rate) won't help. And it might again manifest itself in a zig-zag pattern. If batch size is one of the parameters try to decrease it.

The least likely issue might be the encoding architecture of your network. And especially the modest number of 8 neurons on the last layers. In this case individual neurons might have a considerable impact on the final output. And even little adjustments resulting from a single step of back propagation could potentially flip the sign of that neuron's activation value, impacting the results of other examples. Try increasing the number of neurons on the last layers. I'd personally suggest trying an architecture of 50x50x50.

Hope this helps!

  • $\begingroup$ Sounds great. Thank you for your advice. I use a learning rate of 0.01 with expo. decay 0.96 every 1k steps. I tried also 0.1 and 0.001 as starrting values but could not get better. This brings me to your second point: I use all values, no batches. I.e. a total of 400k values. If I get you right, you say to introduce batch sizes (e.g. 100) to avoid averaging. $\endgroup$ – Frank Apr 29 '16 at 14:25
  • $\begingroup$ You suggest 50x50x50. Some others say descending, some discuss increasing number of neurons. I am little puzzled because at the moment I feel like being in the monkey business and not following a high-tech procedure, i.e. fully deductive :-) $\endgroup$ – Frank Apr 29 '16 at 14:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.