Neural Network Loss Function - Mean Square Error: questions about what 'n' signifies

I can't understand how the answers in this question answered the question. please help me to understand the following case:

Lets look at an output layer with 10 neurons. the label/target is also a vector of the size 10. and lets say we have only 2 samples/instances.

For the first sample we get:

output layer: [0.9, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1]

lable/target: [1 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 ]

and for the second sample:

output layer: [0.2, 0.9, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2]

label/target: [0 , 1 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 ]

The way I understood it, we can compute the loss for each sample:

Loss = (1/10)*sum((output_layer - label)**2)

This way , the n represents the number of neurons (10). the answers says the n is the number of samples (2). what is my mistake here?

And also, if I work with full batch and not mini batch, I would want to update the weights after the network went through all the samples, wich means(I think) I want a single loss function for all the samples. The only way I think about is to define the n as the number of neurons in the output layer times the number of samples. Is that how its done?

Loss_for_all_samples = (1/20)*sun((all_outputs - all_labels)**2)

Where all_outputs will be the sum of the output_layer of each sample and all_labels the sum of lables for each sample.


As the answer from 10xAI notes, n in the loss function refers to the number of samples over which you are calculating the loss, meaning that you are basically calculating the average loss for a specific batch of data. Your error is that you are dividing by the number of output neurons, which is incorrect as the number of output neurons/number of classes has no impact on the loss (in the case of MSE). The loss function you are referring to, mean squared loss, is only applicable to regression problems, whereas your example makes use of classes and is therefore a classification problem. In regression problems there is often only one continuous value you're trying to predict (and only one output neuron), there therefore is no need to divide by the number of output neurons. For your example something like the cross-entropy loss would make more sense, and there you are summing the losses over the different classes, but then still divide them by the number of samples to get an average loss for a batch.

  • $\begingroup$ I understand you, thanks. my confusion comes from seeing 3blue1brown's video on gradient descent. <youtube.com/watch?v=IHZwWFHWa-w&t=243s> from minute 3:50. if you can take a look to see what he did there it would be nice. $\endgroup$
    – EB97
    May 30 at 14:07
  • $\begingroup$ oh and also, i know its a classification problem because i need to predict certain number, but why MSE is only applicable for regression? the thoery in the video seems logical $\endgroup$
    – EB97
    May 30 at 14:18
  • $\begingroup$ Quickly had a look at the video, see the text at 4:22 which says "Average cost of all training data", this is what the n refers to in the formula. You are dividing the value by n to get the average loss for a specific number of samples. The difference between the actual value and predicted value at each output neuron (i.e. the loss for each class) is simply added to get the total loss for a single sample, not divided by the number of neurons as in your example. $\endgroup$
    – Oxbowerce
    May 30 at 17:33
  • $\begingroup$ While it is technically possible to use MSE for classification problems, it is not the correct way as it influences the way the gradients are computed. There are quite a few articles you can find on this, for example this post. I assumed MSE was used in the video since it generally is an easy to understand loss function, cross-entropy loss uses logarithms which would maybe require some extra explanation. $\endgroup$
    – Oxbowerce
    May 30 at 17:33
  • $\begingroup$ so you have any idea why he sums the error squared of each neuron? $\endgroup$
    – EB97
    May 30 at 21:32

If you're looking at how the MSE is implemented in, e.g. Python, you'll see that, in essence, the mean is taken across both dimensions, i.e. features and samples:

def mse(x, y):
    diff = x - y
    err = np.square(diff)
    return np.mean(err)

(when no axis is given as a parameter to np.mean, NumPy takes the mean across all axes)

See, for example, also here and the scikit learn implementation. Applying above function to a multi-dimensional example like yours above with 10 features and 2 samples (i.e. inputs have (2,10) as a shape) gives:

y_pred = np.array([
    [0.9, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1],
    [0.2, 0.9, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2]])

y_true = np.array([
    [1  ,  0 ,  0 , 0  , 0  , 0  , 0  , 0  , 0  , 0  ],
    [0  ,  1 ,  0 , 0  , 0  , 0  , 0  , 0  , 0  , 0  ]])

mse(y_true, y_pred)

OUT: 0.02350000000000001

In general,
when we discuss this topic, it is around the idea of Stochastic GD, Mini-batch GD, Batch GD. The idea of averaging is to move the gradient towards an average of the Batch.
So, "N" refers to the batch_size in general.

On aggregation at the Output layer, it is more of a design implementation I believe.
Check this example for Keras

import tensorflow as tf, numpy as np
from tensorflow.python.keras.layers import Input, Dense
from tensorflow.python.keras.models import Model, Sequential
import keras.backend as K

model = Sequential()
model.add(Dense(3, activation="relu", input_shape=(2,), kernel_initializer=tf.keras.initializers.Ones()))
model.add(Dense(2, activation="relu", kernel_initializer=tf.keras.initializers.Ones()))

x = tf.constant([[2.,2.],[4.,4.],[5.,5.]])
y =tf.constant([[10.,12.], [20.,20.], [25.,25.]])

 # This is Keras standard definition i.e. reduce to mean on last axis
def mse_loss(y_true, y_pred): return K.mean(K.square(y_pred - y_true), axis=-1)

from keras.callbacks import LambdaCallback

def batchOutput(batch, logs): print("Finished batch: " + str(batch), end=''); print("--->",logs)
batchLogCallback = LambdaCallback(on_batch_end=batchOutput)

model.fit(x,y, batch_size=1, epochs=1,callbacks=[batchLogCallback], verbose=0, shuffle=False)

Finished batch: 0---> {'loss': 2.0}
Finished batch: 1---> {'loss': 8.509464263916016}
Finished batch: 2---> {'loss': 12.874275207519531}

You can manually calculate the loss as the input and the weights have been taken as simple integers.
The final loss [4, 0] is averaged to 2.

def mse_loss(y_true, y_pred): return K.sum(K.square(y_pred - y_true), axis=-1)

If you use the above function output will be,

Finished batch: 0---> {'loss': 4.0}
Finished batch: 1---> {'loss': 17.01892852783203}
Finished batch: 2---> {'loss': 25.748550415039062}


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.