0
$\begingroup$

I notice that most of my successfully trained CNNs initiate (i.e. first epoch) loss is in the 1-2 range. Typically, these are image classifiers, either for discrete classes (cat, dog, ship, etc) or semantic (pixelwise) segmentation. Occasionally, either because of poor design or architecture misunderstand on my part, I see an initial loss in the 60s; the other day one started at 200 (and didn't converge).

So my question is: is there a heuristic that would allow me to determine a ball park value for my initial loss value? If so, on what aspects of my trained model is it contingent?

$\endgroup$
1
$\begingroup$

The initial loss is purely dependent on your weight initialization and your data normalization. Random weights = random initial loss. If you're interested in the variance of the loss, this will roughly relate to the number of layers you have and number of weights in each layer. For a good example, check out this paper:

Understanding the difficulty of training deep feedforward neural networks -- Glorot, Bengio

$\endgroup$
1
$\begingroup$

In the Stanford CS231n coursework, Andrej Karpathy suggests the following:

Look for correct loss at chance performance. Make sure you’re getting the loss you expect when you initialize with small parameters. It’s best to first check the data loss alone (so set regularization strength to zero). For example, for CIFAR-10 with a Softmax classifier we would expect the initial loss to be 2.302, because we expect a diffuse probability of 0.1 for each class (since there are 10 classes), and Softmax loss is the negative log probability of the correct class so: -ln(0.1) = 2.302.

$\endgroup$
1
  • $\begingroup$ What if you have a sigmoid output layer, and binary crossentropy loss? $\endgroup$ Feb 11 '20 at 19:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.