1
$\begingroup$

I have been trying to attempt plant disease detection using transfer learning methods. I chose ResNet50 first. I also performed a baseline model which is a CNN model. In resnet50, I used cross entropy loss and trained the model for 30 epochs. I did batch normalization too. Initially epoch 1 loss was 112.5250 and the training loss was 87.512. But, for the last epoch it was 2.1660 and Validation loss was 1.8905 with Validation accuracy as 0.995. The overall accuracy of the model was 98.8% and the model doesn't seem to overfit too.

Before training the model, I performed hyper parameter optimization where I optimized Learning rate and momentum using Bayesian optimization. I optimized weight decay using Cross validation. I performed batch normalization. Before that, I trained the model without any optimization. I just assumed the learning rate to be 0.001, same loss function and trained for 30 epochs. In this case, the model started with a loss value of 114.2 and validation loss was 130.46. It converged to 9.9038 with accuracy of 97%.

So, my question is, it is possible for the model to start with such a high loss value despite giving a good accuracy at the end? Does the magnitude of the loss have nothing to do with the correctness of the model nor the accuracy? If the model started with a huge loss, but gave a good accuracy, is my model bad?

First 5 epochs

EPOCH: 1 

Epoch time taken: 28.61398434638977 seconds. Epoch 1 loss: 118.5250 Validation loss: 87.5214 Validation accuracy: 0.753

EPOCH: 2 

Epoch time taken: 28.737553358078003 seconds. Epoch 2 loss: 68.8141 Validation loss: 50.3452 Validation accuracy: 0.858

EPOCH: 3 

Epoch time taken: 28.45273518562317 seconds. Epoch 3 loss: 41.5181 Validation loss: 31.1520 Validation accuracy: 0.928

EPOCH: 4 

Epoch time taken: 28.434630155563354 seconds. Epoch 4 loss: 26.8844 Validation loss: 21.1203 Validation accuracy: 0.948

EPOCH: 5 

Epoch time taken: 28.762181282043457 seconds. Epoch 5 loss: 19.3646 Validation loss: 15.4843 Validation accuracy: 0.955

Last 5 epochs

EPOCH: 26 

Epoch time taken: 28.35645818710327 seconds. Epoch 26 loss: 2.8168 Validation loss: 2.2771 Validation accuracy: 0.990

EPOCH: 27 

Epoch time taken: 28.51978588104248 seconds. Epoch 27 loss: 3.3986 Validation loss: 2.2274 Validation accuracy: 0.992

EPOCH: 28 

Epoch time taken: 28.51390767097473 seconds. Epoch 28 loss: 2.2707 Validation loss: 1.9976 Validation accuracy: 0.992

EPOCH: 29 

Epoch time taken: 28.456573247909546 seconds. Epoch 29 loss: 2.5344 Validation loss: 1.9272 Validation accuracy: 0.990

EPOCH: 30 

Epoch time taken: 28.486974239349365 seconds. Epoch 30 loss: 2.1660 Validation loss: 1.8905 Validation accuracy: 0.995
```
$\endgroup$
2
  • $\begingroup$ When the training is initiated, the parameters of the model are randomly initialized i.e. the filters of the Convolution layer and the biases, and hence the initial loss will be any arbitrary number. Training the model for a 2nd time, will again yield a different initial loss, as the parameters were different again ( randomly initialized ). If the initial loss is high, we can either use a larger learning rate for faster convergence. $\endgroup$ May 22, 2021 at 7:45
  • $\begingroup$ @ShubhamPanchal Okay. Maybe I will try to increase the learning rate. But I am afraid that it could result in overfitting of the model. Should I increase the number of epochs so that I can the loss converges even more ? Will that be an improvement ? $\endgroup$
    – Anu Deepa
    May 22, 2021 at 9:33

1 Answer 1

1
$\begingroup$

The value of the loss in the first epochs is irrelevant. The network weights are initialized at random, so it is perfectly expected that the behavior of the network differs from the desired one.

However, this does not mean that you should only look at the last value of the training and validation loss. Instead, you should plot the whole curves, to understand the training dynamics. From the curves, you can see whether your network overfitted or underfitted, if there's still room for improvement if you train the model for longer, etc.

There are many questions in this site asking about how to interpret training/validation curves (e.g. this and this). You may take a look at them to understand how to interpret yours. From the values you posted, it seems there's still room for improvement if you train for more epochs.

$\endgroup$
2
  • $\begingroup$ I plotted the curves. The accuracy curve has converged and the validation loss and training loss curves has converged too in my opinion.... but I will take your suggestion for training it for more epochs. Maybe I could get some better insights from that. Thanks $\endgroup$
    – Anu Deepa
    May 22, 2021 at 9:30
  • $\begingroup$ Please, consider marking the answer as correct if it was helpful. $\endgroup$
    – noe
    May 22, 2021 at 10:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.