How do I know what the best number of layers is required to achieve the highest accuracy

Question

I'm learning from Udacity using this video.

I saw this piece of code:

model = nn.Sequential(nn.Linear(784, 128),
                      nn.ReLU(),
                      nn.Linear(128, 64),
                      nn.ReLU(),
                    #   nn.Linear(64, 32),
                    #   nn.ReLU(),
                      nn.Linear(64, 10),
                      nn.LogSoftmax(dim=1))

criterion = nn.NLLLoss()
optimizer = optim.SGD(model.parameters(), lr=0.003)

epochs = 5
for e in range(epochs):
    running_loss = 0
    for images, labels in trainloader:
        # Flatten MNIST images into a 784 long vector
        images = images.view(images.shape[0], -1)
    
        # TODO: Training pass
        optimizer.zero_grad()

        output = model(images)
        loss = criterion(output, labels)
        loss.backward()
        optimizer.step()

        if e == 1 and running_loss == 0:
            print(loss)

        running_loss += loss.item()
    else:
        print(f"Training loss: {running_loss/len(trainloader)}")

Then, I was playing around with layers and neurons.

I have tried:

1. Adding 1 more layer (64, 32) between (128, 64) and (64, 10), and changing the (64, 10) to (32, 10), increasing the epochs.
2. Replacing the (128, 64) with (128, 32), increasing the epochs
3. Adding 1 layer (784, 256), (256, 128), increasing the epochs

Etc.

From what I saw here, they can all achieve good accuracy with enough epochs of training.

Then, my question is how can I find the best number of layers and neurons in each layer (architecture) which can:

1. Achieve the highest accuracy
2. Still be the simplest model (least number of parameters)

Answer 1

The number of layers and the number of nodes in each layer are model hyperparameters that you must specify. So, we cannot directly say this no of hidden layers / this no of hidden units / this learning rate .. are better. But we can use some techniques :

• Intuition:

  So based on the complexity of your model, you need to choose a value that you feel suits it the most.

• Experiment:

  Now, if your intuition fails, then you should experiment and see the nature of the model. So, try to plot the cost function for different values of the hyperparameter and understand the behavior of the model, with which you can probably get the best value for your hyperparameter.

Generally, if your model is expected to learn complex patterns, then you should go for depth.

For no of neurons in a layer, there are some rule-of-thumb methods:

• The number of neurons should be between the size of the input layer and the size of the output layer. (or mean of both)
• The number of neurons should be : (2/3 the size of the input layer + the size of the output layer)

There are some other rules and formulas, but I think these rules should give you a good start. But still, the key is your intuition and experimentation.