I need help with understanding the LeNet-5 CNN:

  1. How/Why does FC3 and FC4 have 120 and 84 parameters?
  2. How are the filters 6 and 16 chosen? (intuition based on the dataset?)

enter image description here

Everywhere that I have looked, I haven't found an answer to #1, including LeCunn's original paper.

What am I missing?

I am tasked with swapping out the 5 x 5 kernels (f = 5), with 3 x 3 kernels (f = 3). If I understand where those values (especially 84, 120) come from, I think I will be able to do it. I was able to implement LeNet-5 using PyTorch.

If you have any suggestions what values would work best and why, I would be grateful. The dataset is Cifar-10.


I modified my code, as @Oxbowerce suggested to make to sure the the image size before unflattening matches the image size in the view before activating the first fully connected network:

In my constructor for the class LeNet:

self.fc1 = nn.Linear(4 * kernel_size * kernel_size * 16, 120)  # added 4

In the feed forward network:

x = x.view(-1, 4 * self.kernel_size * self.kernel_size * 16) # added 4

Here is my network class:

class LeNet(nn.Module):

def __init__(self, activation, kernel_size:int = 5):

    self.kernel_size = kernel_size

    self.conv1 = nn.Conv2d(in_channels=3, out_channels=6, kernel_size=kernel_size, stride=1)
    self.pool = nn.MaxPool2d(kernel_size=2, stride=2)
    self.conv2 = nn.Conv2d(in_channels=6, out_channels=16, kernel_size=kernel_size, stride=1)

    self.fc1 = nn.Linear(4 * kernel_size * kernel_size * 16, 120)  # added 4
    self.fc2 = nn.Linear(120, 84) 
    self.fc3 = nn.Linear(84, 10)

    self.activation = activation

def forward(self, x):

    x = self.pool(self.activation(self.conv1(x)))
    x = self.pool(self.activation(self.conv2(x)))

    x = x.view(-1, 4 * self.kernel_size * self.kernel_size * 16) # added 4

    x = self.activation(self.fc1(x))
    x = self.activation(self.fc2(x))
    x = self.fc3(x)

    return x

I call the train method with this:

model.train(activation=nn.Tanh(), learn_rate=0.001, epochs=10, kernel_size=3)

Train method:

def train(self, activation:Any, learn_rate:float, epochs:int, momentum:float = 0.0, kernel_size:int = 5) -> None:

    self.activation = activation
    self.learn_rate = learn_rate
    self.epochs = epochs
    self.momentum = momentum

    self.model = LeNet(activation=activation, kernel_size=kernel_size)

    optimizer = torch.optim.SGD(params=self.model.parameters(), lr=learn_rate, momentum=momentum)

    for epoch in range(1, epochs + 1):
        loss = 0.0
        correct = 0
        total = 0
        predicted = 0

        for batch_id, (images, labels) in enumerate(self.train_loader):
            images, labels = images.to(device), labels.to(device)

            outputs = self.model(images)
            # Calculate accurracy
            predicted = torch.argmax(outputs, 1)
            correct += (predicted == labels).sum().item() # <--- exception
            total += labels.size(0)

        accuracy = 100 * correct / total


    for i in self.model.parameters():

Here is the stack trace of the error:

Traceback (most recent call last):
  File "/usr/lib/python3.9/unittest/case.py", line 59, in testPartExecutor
  File "/usr/lib/python3.9/unittest/case.py", line 593, in run
  File "/usr/lib/python3.9/unittest/case.py", line 550, in _callTestMethod
  File "/home/steve/workspace_psu/cs510nlp/hw2/venv/lib/python3.9/site-packages/nose/case.py", line 198, in runTest
  File "/home/steve/workspace_psu/cs510dl/hw2/test_cs510dl_hw2.py", line 390, in test_part2_relu_cel_k3
    model.train(activation=activation, learn_rate=learn_rate, momentum=momentum, epochs=epochs, kernel_size=kernel_size)
  File "/home/steve/workspace_psu/cs510dl/hw2/cs510dl_hw2.py", line 389, in train
    correct += (predicted == labels).sum().item()
  File "/home/steve/workspace_psu/cs510nlp/hw2/venv/lib/python3.9/site-packages/torch/tensor.py", line 27, in wrapped
    return f(*args, **kwargs)
Exception: The size of tensor a (16) must match the size of tensor b (4) at non-singleton dimension 0

The choice for the number of neurons in the last two dense layers and the number of filter is somewhat arbitrary and most of the time determined by trying different configurations (using something like a hyperparameter grid search). See also this answer on stats stackexchange. If you want to change the size of the 5x5 kernels you will only have to change the number of neurons in the first fully connected layer, the last two don't have to change for the network to be "valid".

  • $\begingroup$ Thank you @Oxbowerce. I added my code above. Shouldn't I change all kernel sizes to 3, except for the pooling layers? Sorry, if this is not the place to post code, I'll ask on stack overflow. $\endgroup$
    – Steve3p0
    Feb 8 at 11:57
  • 1
    $\begingroup$ It depends on for which layers you want to change the kernel size, if it just for the convolutional layers you would only have to change the kernel sizes for self.conv1 and self.conv2. $\endgroup$
    – Oxbowerce
    Feb 8 at 12:34
  • $\begingroup$ If I wanted to change just the convolutional layers and not the pooling layers, I would need to change the four lines above and make sure they are using the kernel_size param, right? $\endgroup$
    – Steve3p0
    Feb 8 at 12:54
  • 1
    $\begingroup$ You'd need to change the first two lines to the kernel size you want to use, the number of neurons in your fully connected layer depends on tensor size and not on the kernel size (at least not directly, in this example the kernel size and image size happen to be equal). $\endgroup$
    – Oxbowerce
    Feb 8 at 13:17
  • 1
    $\begingroup$ They do not need to be equal, it is just that in this case they are equal (i.e. image size before unflattening is 5x5 and the kernel size for the convolutional layers is also 5x5). That is why using kernel_size to calculate the number of neurons in the first fully connected layer works, this won't work if you change the kernel_size to 3. $\endgroup$
    – Oxbowerce
    Feb 8 at 14:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.