5
$\begingroup$

are there any resources out there (book, blog, your own answer post etc.) that gives advise on modelling strategy of deep neural net?

I know how to fit a neural net, I know how to change settings like number of hidden layers, activation function, dropout etc. I know how to use cross-validation to validate models.

But what I need is advise on the actual modelling process. i.e. given a dataset (which has been cleaned and explored), where do you start? What type of neural net do you train first? How do you then tune it?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

I cannot recommend highly enough this online book on neural networks.

The tricky part about neural networks is the stuff you mentioned -- what value to use for the learning rate, what topology to use for the network, etc etc -- we call these things hyperparameters to distinguish from parameters which are estimated by the optimization process.

This big section of the aforementioned book covers exactly that. I don't know if you'll find what you want though. The reason is because there is no silver bullet.

Improve this answer
$\endgroup$
0
$\begingroup$

There should be a distinction between the terms of neural networks and deep neural networks. I found this link: https://www.quora.com/How-does-deep-learning-work-and-how-is-it-different-from-normal-neural-networks-applied-with-SVM.

Unfortunately, there is no general rule and probably will never be one that says that you should use a certain type of network "a priori". It depends on the task you are trying to solve, but you already knew that.

I suppose you are wondering what are the meanings of the additional hidden layers, how to figure out how many of them should be there and how large they should be. Also, how the additional parameter affects this network and how to think through your model when something doesn't work right.

Again, it depends. There are lots of types of networks that are good for additional problems. For example, LSTMs are good when working with series of data. If you want to tune your LSTM to work well, you should go INTO it and figure out how EXACTLY does it work.

I highly recommend this playground site for neural networks: http://playground.tensorflow.org

I also suggest https://www.tensorflow.org/versions/r0.9/how_tos/graph_viz/index.html to inspect your code's architecture. Additionally, try to explore the TensorFlow documentation.

In the end, I advise you to read. Read more. (http://karpathy.github.io/)

Improve this answer
$\endgroup$
2
  • $\begingroup$ «"ordinary" neural networks for supervised and deep neural networks for unsupervised tasks.» - I am sorry but your comment is just wrong. Deep neural networks were a set of techniques that were discovered to overcome the vanishing gradient problem which was severely limiting the depth of neural networks. Deep neural networks can be used for both unsupervised (e.g. auto-encoders) or supervised (e.g. convolutional neural networks) problems. $\endgroup$
    – Ricardo Magalhães Cruz
    Jun 15, 2016 at 11:26
  • $\begingroup$ I agree. That really is wrong, I removed that part and left the link. $\endgroup$
    – Kristijan
    Jun 15, 2016 at 16:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.