Guiding principle of Neural Network structure building

Question

I study Neural Networks and I pretty much understand the logic of the structure with layers, activation functions and connections. But a fundamental question is not clear: how does one put together the actual structure from the several possible combinations? It seems me that most people just combine these quite randomly without any proper reason (or with brute force "grid search" in better cases).

I've read several tutorials but nobody explained the reason of choosing "sigmoind" activation function over "tanh" for instance, not to mention which one to pick in case of different activation functions in different layers. I stress that I understand the working of these functions by themselves, but the logic of the order of them is quite a puzzle.

Answer 1

You would use a sigmoid if you want your activations to be between zero and 1 and tanh between -1 and 1, this is important for the final layer, but for a hidden layer there is not much difference between them.

Here is an article on how to build a neural network in a more systematic way.

The basic principle is to start with only one layer, and make it bigger and bigger until you see satisfactory results, if you don't, add another layer and start over. Oh and make sure you use batch normalization, and don't use sigmoids or tanh's, use PreLUs or ELUs: https://www.linkedin.com/pulse/keras-neural-networks-win-nvidia-titan-x-abhishek-thakur?trk=prof-post

In general:

Sigmoids: use for output layer which are probabilities with logloss loss function, and for gates in RNNs.

Tanh: use for updating the state in RNNs (linear-like activations might make it unstable), or for an output layer with variables between -1 and 1.

Rectifiers: use for activations in hidden layers of feed-forward network.

Answer 2

Theory is still way behind practice in neural nets. But a classic paper by Yann LeCun has some helpful guidance about architectural choices including activations and scaling.

For those coming by this question who might not know: tanh is just a shifted and scaled sigmoid; see this question.

Answer 3

You're right that most of the structure has to come from trial and error on your dataset. However, there are tips that can save you time, although that's not always the case. Rectified linear units are popular these days because they're more resistant to the vanishing gradient problem compared to sigmoid activation functions. I don't see a big difference between a sigmoid and tanh activation function though. Another helpful tip is to use convolutions for 2D (ex image) datasets. You also don't want to add more layers than the size of your dataset can handle (you might overfit). Ultimately these are all just suggestions. No one will be able to say for sure, ahead of time, what will work on your particular problem.

Answer 4

There is a good deal of trial-and-error when building NNs, but here's a bit of semi-structured advice on the matter.

training networks in general:

In addition to the LeCun paper mentioned here, Geoffry Hinton has a great guide to the overall process here, which i suggest you read. You can gain some intuition about batch training sizes as well, yet another thing to consider when training NNs.

Hidden layers

Regarding hidden layers on non-recurrent/non-convolutional networks with $n$ features, I've found that a single hidden layer of size $x$ somewhere in the range of $n > x > \log(n)$ works well.

ReLUs

Using a ReLU in the hidden layer helps with sparsity and to combat vanishing gradients. ReLUs have an output of $[0, \infty]$ as it is $f(x) = max(0, x)$ where $x$ is the unit's input, where sigmoid units have outputs in the range $[0, 1]$. There is significantly more biological plausibility for ReLUs, as they better approximate a common neuroscience model for neuronal firing, the leaky-integrate-and-fire model. They are slightly more efficient computationally (the only operations are compare, add,and multiply). For even more detail about ReLUs, read Glorot, Bordes, and Bengio.