CNN Back Propagation without Sigmoid Derivative

Question

I'm new to CNN and trying to study some MATLAB sample codes (cause I need to know the internal calculation). I recently realized that the sample code I'm using doesn't multiply error by sigmoid's derivative in back propagation. The feed forward process has sigmoid as last layer's activation function so from my understanding, back propagation error = (outputs - target) * sigmoid's derivative(outputs). However, the author intentionally disabled this multiplication with the following code:

if cnn.loss_func == 'cros' 
    if cnn.layers{cnn.no_of_layers}.act_func == 'soft'
        cnn.CalcLastLayerActDerivative = 0;
    elseif cnn.layers{cnn.no_of_layers}.act_func == 'sigm'
        cnn.CalcLastLayerActDerivative = 0;
    end    
end

My reference code.

When cnn.CalcLastLayerActDerivative = 0, error is defined just as (outputs - target). I tried to initialize cnn.CalcLastLayerActDerivative = 1 so that sigmoid's derivative is considered in back propagation but then I got worse error rate. I'm not sure whether it's just because sigmoid's derivative is in the range [0,0.25] or I'm not understanding back propagation correctly. Does anyone know why this is happening and whether I should add sigmoid's derivative in my calculation?

Thanks!

Answer 1

error is defined just as (outputs - target)

This is the correct gradient for cross-entropy loss function with Sigmoid as the last layer.

For squared (quadratic) loss $$(y-f(x))^2,$$ the gradient is, as you said, $$(y-f(x))f'(x)$$ (constant $2$ is removed), but for binary cross-entropy loss $$y\text{log}f(x) + (1-y)\text{log}(1-f(x)),$$the gradient is $$yf'(x)/f(x) - (1-y)f'(x)/(1-f(x)),$$ since for Sigmoid we have $f'(x)=f(x)(1-f(x))$, by substitution the gradient becomes $$y(1-f(x)) - (1-y)f(x)=y-f(x)$$ To distinguish between these two gradients, author sets cnn.CalcLastLayerActDerivative = 0 to be checked later in an if statement in bpcnn.m file as follows (comments don't exist in the original code):

...
else
  % error = (f(x) - y)
  er = ( cnn.layers{cnn.no_of_layers}.outputs - yy);
...
if cnn.CalcLastLayerActDerivative ==1 
    % change the error from (f(x) - y) to f'(x)(f(x) - y)
    er =applyactfunccnn(cnn.layers{cnn.no_of_layers}.outputs,cnn.layers{cnn.no_of_layers}.act_func, 1, er);
end

which means gradient is $(y-f(x))f'(x)$ for quad and $(y-f(x))$ for cros (bad variable name!).

As a side note, author only allows Sigmoid for cross entropy which means only binary classifier is supported (multi-class classifier requires SoftMax).

error('cross entropy is implemented only when last layer is sigmoid');

EDIT

Thanks to @Edison for pointing out that error and gradient were not handled the same as loss values in the code, which substantially changed the final answer.

Answer 2

Thank you(Esmailian) so much for your answer. I agree with you that the author distinguished the two losses by the setting cnn.CalcLastLayerActDerivative=0/1.

However, in the original codes, the calculation of gradient for corss-entropy: yf′(x)/f(x)−(1−y)f′(x)/(1−f(x)) is not provided in bpcnn.m. Only the corss-entropy error ylogf(x)+(1−y)log(1−f(x)) is provided but sent to er1 only for plotting the losses:

>     if cnn.loss_func == 'cros' %cross_entropy'
>             if cnn.layers{cnn.no_of_layers}.act_func == 'sigm'
>                 er1 = -1.*sum((yy.*log(cnn.layers{cnn.no_of_layers}.outputs) + (1-yy).*log(1-cnn.layers{cnn.no_of_layers}.outputs)), 1);
>             else
>      ...
>             end
>             cnn.loss = sum(er1(:))/size(er1,2); %loss over all examples
>         
>         else
>             er1 = er.^2;
>             cnn.loss = sum(er1(:))/(2*size(er1,2)); %loss over all examples
>          
>     end

Thus, could you provide more detailed answer regarding to this?

Thanks to @Esmailian! All the questions I had are now resolved.