I'm implementing a ResNet network from scratch using PyTorch. This network is unique to my requirements, since I need to perform Image Classification for Satellite Imagery with 14 different channels and of dimensions 8x8 pixels. My training data consists of 588 training images for 7 different classes

However, my issue is that during training, the Training Loss starts massively increasing right after the first epoch, and then reaches Infinity within 10 epochs, and my accuracy is stuck at 0.17.

The initial data transformation is to resize the 8x8 pixel images to 64x64 pixel images.

My ResNet network consists of:

  1. One initial convolution block - An initial layer to reduce image dimensions while increasing channel size.

  2. Three Residual Blocks - These Residual Blocks have a Bottleneck structure (1x1 with x/2 channels, 3x3 with x/2 channels, 1x1 with x channels). Here, the first 2 Blocks have the same structure, after which I shrink the image dimensions while doubling the number of channels and then feed it to the third Residual Block. All my Residual Blocks have ReLU layers, BN layers in between Convolution operations, as well as a Dropout Layer at the end of each Residual Block.

  3. Fully Connected Layer - This consists of a Flattening layer, followed by 2 consecutive Linear Layers separated by a ReLU layer. The final layer outputs class probabilities.

My question is - what could be the possible reasons for such spikes in Training Loss in a ResNet network? For instance, would too few Residual Blocks cause such an issue? Or is it the small size of the dataset (588 images) that might be responsible? Below is the "forward" of my ResNet class object (I can provide the rest of the code also, if needed):

def forward(self, x, conv_channels):

        # Initial Convolution Block
        self.conv1Out = self.conv1block(x)
        # Residual Block 1
        self.conv2Out_1 = self.conv2block(self.conv1Out)
        self.res1 = self.conv2Out_1 + self.conv1Out
        # Residual Block 2 - here, the summation happens after the resizing
        self.conv2Out_2 = self.conv2block(self.res1)
        self.adjBlock = nn.Conv2d(in_channels=conv_channels,
        self.conv2Out_2 = self.adjBlock(self.conv2Out_2)                                 
        self.res1_adj = self.adjBlock(self.res1)
        self.res2 = self.conv2Out_2 + self.res1_adj
        # Residual Block 3
        self.conv3Out = self.conv3block(self.res2)
        self.res3 = self.conv3Out + self.res2
        # Fully Connected Layer
        self.final = self.linear_block(self.res3)
        return self.final
  • $\begingroup$ Maybe an issue in the training loop or optimization hyperparameters ? $\endgroup$
    – Lelouch
    Nov 7, 2023 at 9:38
  • $\begingroup$ Then maybe model became very complex when compared to data. Try to increase weight_decay value. $\endgroup$
    – citara3996
    Nov 10, 2023 at 9:11

1 Answer 1


Did you check if gradients are exploding?I think that would be one reason. Then try gradient clipping.

  • $\begingroup$ I didn't consider that possibility - so I tried clipping the gradients using Pytorch's clip_grad_norm function. However, while that reduced the training loss values, the accuracy is still stuck at around 0.17 over multiple epochs. $\endgroup$ Nov 4, 2023 at 10:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.