0
$\begingroup$

I'm learning from Udacity using this video.

I saw this piece of code:

model = nn.Sequential(nn.Linear(784, 128),
                      nn.ReLU(),
                      nn.Linear(128, 64),
                      nn.ReLU(),
                    #   nn.Linear(64, 32),
                    #   nn.ReLU(),
                      nn.Linear(64, 10),
                      nn.LogSoftmax(dim=1))

criterion = nn.NLLLoss()
optimizer = optim.SGD(model.parameters(), lr=0.003)

epochs = 5
for e in range(epochs):
    running_loss = 0
    for images, labels in trainloader:
        # Flatten MNIST images into a 784 long vector
        images = images.view(images.shape[0], -1)
    
        # TODO: Training pass
        optimizer.zero_grad()

        output = model(images)
        loss = criterion(output, labels)
        loss.backward()
        optimizer.step()

        if e == 1 and running_loss == 0:
            print(loss)

        running_loss += loss.item()
    else:
        print(f"Training loss: {running_loss/len(trainloader)}")

Then, I was playing around with layers and neurons.

I have tried:

  1. Adding 1 more layer (64, 32) between (128, 64) and (64, 10), and changing the (64, 10) to (32, 10), increasing the epochs.
  2. Replacing the (128, 64) with (128, 32), increasing the epochs
  3. Adding 1 layer (784, 256), (256, 128), increasing the epochs

Etc.

From what I saw here, they can all achieve good accuracy with enough epochs of training.

Then, my question is how can I find the best number of layers and neurons in each layer (architecture) which can:

  1. Achieve the highest accuracy
  2. Still be the simplest model (least number of parameters)
$\endgroup$
2
$\begingroup$

The number of layers and the number of nodes in each layer are model hyperparameters that you must specify. So, we cannot directly say this no of hidden layers / this no of hidden units / this learning rate .. are better. But we can use some techniques :

  • Intuition:

    So based on the complexity of your model, you need to choose a value that you feel suits it the most.

  • Experiment:

    Now, if your intuition fails, then you should experiment and see the nature of the model. So, try to plot the cost function for different values of the hyperparameter and understand the behavior of the model, with which you can probably get the best value for your hyperparameter.

Generally, if your model is expected to learn complex patterns, then you should go for depth.

For no of neurons in a layer, there are some rule-of-thumb methods:

  • The number of neurons should be between the size of the input layer and the size of the output layer. (or mean of both)
  • The number of neurons should be : (2/3 the size of the input layer + the size of the output layer)

There are some other rules and formulas, but I think these rules should give you a good start. But still, the key is your intuition and experimentation.

| improve this answer | |
$\endgroup$
  • $\begingroup$ thanks for the answer. So in short for complex patterns, go for depth. How about the width(in term of number of neurons for layers)? $\endgroup$ – Franva Jul 31 at 6:27
  • $\begingroup$ thanks for the update. It gives insight to no of neurons in 1 single layer. But why the video makes the number jumps from 784 to 128 straightaway? why not jumps from 784 to 512, then 512 to 256 then 256 to 128, then 64 to 32 and eventually 32 to 10? $\endgroup$ – Franva Jul 31 at 6:57
  • $\begingroup$ Many hidden layers are also sometimes not useful (might cause overfitting). So, I think that was just an example given in the video, but you can try with other values and see which works best. $\endgroup$ – Sai Sreenivas Jul 31 at 9:39
  • 1
    $\begingroup$ cool, so may I conclude that there might be some rules to follow, but intuition and experiment are the best tools which can get you the best architecture? $\endgroup$ – Franva Jul 31 at 11:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.