1
$\begingroup$

Are there any heuristics for deciding on where to start with the number of layers for a neural network? I built one with with 7 layers that has 40 input features and this took around 8 hours to train completely, but now I want to add a bunch of polynomial features which will increase the input size of my network to a few thousand features. Since there are so many features will my network benefit from having more layers? If so, what is a good starting point for the number of features?

Here is some information about my problem: suppose I have a set of usage data from a website. So if a user logs into the website one day their usage statistics during that day is calculated. I am using a few statistical tools to summarize the data and I want to make a prediction about whether or not the user will use the website in the next 60 days. My plan is to try and engineer all the possible monomial quadratic features from this summary and feed this information into a DNN. Without the feature engineering I have roughly 60 different features, but with it I will have at least an addition 3600 features.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ This can depend a lot on your problem domain. Image-recognition networks, for example, tend to do well with relatively deep networks. You can think of this as learning a hierarchy of features, where convolutional layers aggregate pixels into edges, edges to corners, and corners to more complex shapes. Can you tell us more about the task at hand? $\endgroup$
    – Nuclear Hoagie
    Nov 14, 2017 at 20:26
  • $\begingroup$ Sure! I just updated my post. $\endgroup$
    – user184074
    Nov 14, 2017 at 20:42
  • $\begingroup$ Did you try any simpler models first? $\endgroup$
    – Paul
    Nov 15, 2017 at 1:14

3 Answers 3

Reset to default
3
$\begingroup$

Answer

Are there any heuristics for deciding on where to start with the number of layers for a neural network?

No there isn't one in general. The architecture of the network (including depth) is considered to the new "feature engineering". It's more of an art than science at the moment. Anything above 3 layers worth considering.

I want to add a bunch of polynomial features

You don't need to do this kind of feature engineering like you would in traditional machine learning. If you have a network that is sufficiently wide and deep, your network should be able to "engineer" those features itself. Just think about how the first hidden layer is a linear combination of input features. And the second hidden layers is a linear combination of the first hidden layer. It should be obvious to see how a neural network is capable of structuring any polynomial features itself.

Suggestion

Try using the original 60 features directly and start with 3 layers. Given the size of the network, the training should take less than 10 minutes on a GPU-enabled machine. Compare your result with your benchmark model (you should always have a benchmark model). Play around with the hidden size and layers, rinse and repeat.

Improve this answer
$\endgroup$
9
  • $\begingroup$ Unfortunately on my GTX 1080 ti it takes over 8 hours to train my model with 7 layers. Do you know why this is happening? $\endgroup$
    – user184074
    Nov 14, 2017 at 22:05
  • $\begingroup$ @user184074 How many samples do you have? And what's the batch size you are using? My machine has a 1080 ti as well, and I was working on a problem with about the same number of features, I can easily fit 10,240 samples in a batch. If it's not your batch size, you need to share your code for us to help you. $\endgroup$
    – Louis T
    Nov 14, 2017 at 22:11
  • $\begingroup$ @user184074 Even if you have crazily large samples size, you will probably see your network converge way before it finishes processing all samples. At that point, you can stop unless you are really satisfied with the overall architecture and just what to squeeze out that final decimal of performance. $\endgroup$
    – Louis T
    Nov 14, 2017 at 22:16
  • $\begingroup$ I have almost a million rows but my batch size is really small. Does this affect performance? Also, I am running 150 epochs. $\endgroup$
    – user184074
    Nov 14, 2017 at 22:18
  • $\begingroup$ @user184074 That's not that big. Only 100 batches if your batch size is 10,000. I suspect you were using a small batch size because you had so many features. $\endgroup$
    – Louis T
    Nov 14, 2017 at 22:21
1
$\begingroup$

Yes, I would also totally agree with Louis T, as he has suggested it is better if you start with 3 hidden layers and see how the model is performing.

There are couple of things which I would like to add to his answer:

I would recommend you to do feature engg. on data so that you can make better sense out of it, rather then model taking over that task, one of the major reason for that is sometimes machine cannot give some meaningful features with respect to the business and the other important thing would be Cost Function, as the model increases its Complexity the more the cost function would be.

So, how do you decide which model would give you better results:

For example, there are 2 networks(assuming using NN for Prediction) :

  1. Net-1: hidden layers = 3 and RMSE = 3.4500
  2. Net-2: hidden layers = 4 and RMSE = 3.4200

If you we look closely RMSE for Net-2 is less but considering the scenario WRT to hidden layers, I would go with Net-1 because we are considering the Complexity and Cost Function for deciding.

Hope my answer is helpful.

NN : Neural Network

RMSE : Root Mean Square Error

Improve this answer
$\endgroup$
2
  • $\begingroup$ Agree. When you have intuitions or domain expertise, by all means do featue engineering. But when comes to unselective polynominal features, its better to let the neural net do the work. $\endgroup$
    – Louis T
    Nov 15, 2017 at 2:13
  • $\begingroup$ Yes true but is it wise to give the model 3600 features ? I think what you said is also right but a bit of exploratory analysis followed by Dimensionality Reduction might be helpful. $\endgroup$
    – Toros91
    Nov 15, 2017 at 2:19
0
$\begingroup$

Don't worry about engineering features for a neural network. The point of them is that they learn the features well themselves.

In a nutshell, first, make sure you have a representative validation set that you check regularly. Then, build a network large enough to overfit, and one small enough to underfit. The sweet spot is somewhere in the middle; find that based on minimisation of validation loss.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.