Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
Can there be some general recommendations for architecture of neural network if there are only a few features, say 2-5 features?
What should be the number of hidden fully connected layers here? How many neurons may be there in each layer? Do number neurons in different layers need to stay constant (e.g. 16,16,16) or rise (e.g. 8,16,32) or fall (32,16,8) or rise and fall (e.g. 16,32,16) or fall and rise (e.g. 32,16,32)?
There's all sorts of rules of thumb for the structure of a neural net (if n features use n+1 or 2n or 1.5n nodes, etc). I wouldn't take any of them as gospel.
The size and structure of your neural network chiefly depends on the complexity of the data you're trying to learn. If your data is already pretty well represented by, say, linear regression, then you don't really need much additional size to your neural network to represent it well. On the other hand, if your five inputs were, say, the move and shoot commands of a recording of a game of Doom, you're gonna need a large neural network to accommodate all the deep and meaningful interactions between the individual commands.
Each successive layer of nodes of a neural network deals with increasingly complicated shapes. The first hidden layer is all sigmoid or ReLU functions, the second layer is combinations of the first layer, the third is combinations of the second layer, etc.
The general structure of the hidden layers is cone-shaped, tapering towards the right. There are lots of nodes in the first layer, and fewer in each successive layer. You can have constant numbers of nodes per layer, though in my playing around it doesn't seem to help much, and many of the layer nodes go relatively unused. The opposite direction (having few first layer nodes and many last layer nodes) is actively harmful, because the later nodes can only ever be combinations of the earlier nodes, and if the first nodes don't create enough shapes for the later nodes to combine, it won't be able to fit anything.
I've provided a couple of interactive examples through tensorflow's playground.
Normal structure (Decreasing nodes per layer)
The first two should converge relatively easily, whereas the third shouldn't converge at all.
