My goal is to properly define a search space for neural architecture search (NAS).

I think a proper definition must handle the following issues.

  1. how to mathematically quantify the topology?
  2. how to define the number in each layer? activation functions?
  • 1
    $\begingroup$ Do you need the mathematical notion of how NN work? You may find them here $\endgroup$ – Shubham Panchal Apr 20 '19 at 2:11
  • $\begingroup$ Besides the above reference, you can also look at the following: A. Pinkus,Approximation theory of the MLP model in neural networks, Acta Numerica (1999), pp. 143--195 (which provides a justification of why using Deep Neural Networks); and S. Shalev-Shwartz and S. Ben-David, Understanding machine learning: from theory to algorithms, Cambridge University Press, 2014. $\endgroup$ – Tuyen May 21 '19 at 4:15

You can use directed graphs. Each neuron is a node in the graph. Each neuron has list of its neighbours and activation function. You can add remove nodes to this graph.

If you just want to search for simple architectures you can use two lists. One for number of neurons in layer and one for activation function of neurons in that layer.

So this is a neural network with 3 layers. First layer has 5 neurons and echo of them have Relu activation function.

[5,3,2] ['Relu', 'sigmoid', 'Relu']

Hope that helps.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.