1
$\begingroup$

For example consider object localization problem. Here NN will have 5 ouputs. output[0] will tell probability of object present in image, other 4 will tell bounding box coordinates. As we see that output[0] has to use classification loss like cross entropy and output[1] to output[4] will have to use regression loss like Mean-squared-error.

So Total loss is something like this:

loss=Cross_entropy(output[0],Y[0])+MSE(output[1:5],Y[1:5]) #Y is true value

Are loss like that backprogationable in vectorised form? Can I implement that kind of loss in tensorflow? If yes, how does tensorflow do that? Does it perform differentiation on each element of vector or matrix instead of whole thing at once?

$\endgroup$
1
$\begingroup$

Yes, these type of loss functions can be optimized using backpropagation, also in Tensorflow.

The value of the loss is a scalar (same as just the cross entropy, or the MSE, otherwise you wouldn't be able to add them), which means that it doesn't really work any different from just optimizing for any other scalar loss function. As long as the operations involved in calculating the loss function are differentiable ("smooth") enough, Tensorflow (or any other framework that does automatic differentation) doesn't care.

Think of it this way: calculating any interesting loss function involves summing up a bunch of terms living in a higher-dimensional vector space. In this case, for some of the directions in this vector space, you apply a different function on them than for others in that vector space. Doesn't really matter to Tensorflow.

What does matter though is how you sum them: since the cross entropy and the MSE are not working on the same type of units (think dimensional analysis), you have to determine some sort of scale between them. You can view this as a hyperparameter, where you can chose how important it is to have the classification correct vs how correct your bounding boxes are.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.