# Why does neural network need loss as scalar?

I have a loss function that's a weighted cross entropy loss for binary classification

def BinaryCrossEntropy_weighted( y_true, y_pred, class_weight ):
y_true= y_true.astype(np.float)
y_pred = K.clip(y_pred, K.epsilon(), 1 - K.epsilon())
first_term = class_weight * (y_true) * K.log(y_pred + K.epsilon())
second_term = class_weight * (1.0 -y_true) * K.log(1.0 - y_pred + K.epsilon())
loss = -K.mean(first_term + second_term, axis=0)
return loss


And when I run this

loss=BinaryCrossEntropy_weighted( np.array(y),np.array(predict), class_weight )


I got output

<tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.16916199])>


If one can observe carefully, can see that the loss is a vector(of dim(1,) ) not a scalar and I was directly passing this loss to my gradient tape and optimizer,

grads1 = tape.gradient(loss, Final_model.trainable_weights)


Result of this was my loss not decreasing over multiple epoch, meaning my model weight was not being updated ,meaning gradient was not able to pass down/not able to calculated, Now am I correct ?

If I am correct, Now the big question is why tensorflow doesn't allow/accept the loss as a vector ? and in general does NN allow loss value as vector ?

This is more a "programming" question rather than a data science one, however, I'll try to clarify some points:

1. loss has to be a scalar since the training procedure is driven by the minimisation of such a function and there is no definition of vector minimisation, nor of order in vector spaces.
2. I imagine you would like an automatic conversion of the quantity
<tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.16916199])>


into a scalar (such that the minimisation can have effect), this can be achieved by

loss = tf.reshape(loss, []).numpy()

• So I am correct that my loss is not decreasing because of this reason, then why tensorflow don't put any error ? Mar 24, 2022 at 11:27
• I cannot say why your loss is not decreasing without seeing the model. I imagine the reshape is done automatically (hence no error raise). Mar 24, 2022 at 16:42