1
$\begingroup$

Can there be some general recommendations for architecture of neural network if there are only a few features, say 2-5 features?

What should be the number of hidden fully connected layers here? How many neurons may be there in each layer? Do number neurons in different layers need to stay constant (e.g. 16,16,16) or rise (e.g. 8,16,32) or fall (32,16,8) or rise and fall (e.g. 16,32,16) or fall and rise (e.g. 32,16,32)?

$\endgroup$

1 Answer 1

1
$\begingroup$

There's all sorts of rules of thumb for the structure of a neural net (if n features use n+1 or 2n or 1.5n nodes, etc). I wouldn't take any of them as gospel.

The size and structure of your neural network chiefly depends on the complexity of the data you're trying to learn. If your data is already pretty well represented by, say, linear regression, then you don't really need much additional size to your neural network to represent it well. On the other hand, if your five inputs were, say, the move and shoot commands of a recording of a game of Doom, you're gonna need a large neural network to accommodate all the deep and meaningful interactions between the individual commands.

Each successive layer of nodes of a neural network deals with increasingly complicated shapes. The first hidden layer is all sigmoid or ReLU functions, the second layer is combinations of the first layer, the third is combinations of the second layer, etc.

The general structure of the hidden layers is cone-shaped, tapering towards the right. There are lots of nodes in the first layer, and fewer in each successive layer. You can have constant numbers of nodes per layer, though in my playing around it doesn't seem to help much, and many of the layer nodes go relatively unused. The opposite direction (having few first layer nodes and many last layer nodes) is actively harmful, because the later nodes can only ever be combinations of the earlier nodes, and if the first nodes don't create enough shapes for the later nodes to combine, it won't be able to fit anything.

I've provided a couple of interactive examples through tensorflow's playground.

Normal structure (Decreasing nodes per layer)

Same no. of nodes per layer

Rising no. of nodes per layer

The first two should converge relatively easily, whereas the third shouldn't converge at all.

$\endgroup$
6
  • $\begingroup$ Thanks for a good answer. Any comment on role of additional layers like Dropout or BatchNormalization when features are few? $\endgroup$
    – rnso
    Oct 15, 2018 at 4:13
  • $\begingroup$ I don't know much about either of those techniques sorry. $\endgroup$
    – Ingolifs
    Oct 15, 2018 at 4:25
  • $\begingroup$ Last question: any harm in having more neurons even if data is simple? For example, how will an architecture of 32:16:8 (neurons in 3 layers) generally perform for 2-5 feature data? $\endgroup$
    – rnso
    Oct 15, 2018 at 4:28
  • $\begingroup$ Training a neural network is functionally the same as finding local minima in a high-dimensional space. Your proposed nn (32:16:8) for 5 features and 1 output results in (5+1)*32 + (32+1)*16 +(16+1)*8 + (8+1) = 667 neurons, which is 667 dimensions the algorithm must search through to find minima. It takes a lot of computational muscle to do that on a large data set. Always start small and work your way up when the small NN isn't performing adequately. $\endgroup$
    – Ingolifs
    Oct 15, 2018 at 4:39
  • $\begingroup$ It is interesting to note that default number of neurons in Sklearn's MLP is 100 scikit-learn.org/stable/modules/generated/… $\endgroup$
    – rnso
    Oct 15, 2018 at 7:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.