what n represents in the MSE loss function?

Question

Neural Network Loss Function - Mean Square Error: questions about what 'n' signifies

I can't understand how the answers in this question answered the question. please help me to understand the following case:

Lets look at an output layer with 10 neurons. the label/target is also a vector of the size 10. and lets say we have only 2 samples/instances.

For the first sample we get:

output layer: [0.9, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1]

lable/target: [1 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 ]

and for the second sample:

output layer: [0.2, 0.9, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2]

label/target: [0 , 1 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 ]

The way I understood it, we can compute the loss for each sample:

Loss = (1/10)*sum((output_layer - label)**2)

This way , the n represents the number of neurons (10). the answers says the n is the number of samples (2). what is my mistake here?

And also, if I work with full batch and not mini batch, I would want to update the weights after the network went through all the samples, wich means(I think) I want a single loss function for all the samples. The only way I think about is to define the n as the number of neurons in the output layer times the number of samples. Is that how its done?

Loss_for_all_samples = (1/20)*sun((all_outputs - all_labels)**2)

Where all_outputs will be the sum of the output_layer of each sample and all_labels the sum of lables for each sample.

Answer 1

As the answer from 10xAI notes, n in the loss function refers to the number of samples over which you are calculating the loss, meaning that you are basically calculating the average loss for a specific batch of data. Your error is that you are dividing by the number of output neurons, which is incorrect as the number of output neurons/number of classes has no impact on the loss (in the case of MSE). The loss function you are referring to, mean squared loss, is only applicable to regression problems, whereas your example makes use of classes and is therefore a classification problem. In regression problems there is often only one continuous value you're trying to predict (and only one output neuron), there therefore is no need to divide by the number of output neurons. For your example something like the cross-entropy loss would make more sense, and there you are summing the losses over the different classes, but then still divide them by the number of samples to get an average loss for a batch.

Answer 2

If you're looking at how the MSE is implemented in, e.g. Python, you'll see that, in essence, the mean is taken across both dimensions, i.e. features and samples:

def mse(x, y):
    diff = x - y
    err = np.square(diff)
    return np.mean(err)

(when no axis is given as a parameter to np.mean, NumPy takes the mean across all axes)

See, for example, also here and the scikit learn implementation. Applying above function to a multi-dimensional example like yours above with 10 features and 2 samples (i.e. inputs have (2,10) as a shape) gives:

IN:
y_pred = np.array([
    [0.9, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1],
    [0.2, 0.9, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2]])

y_true = np.array([
    [1  ,  0 ,  0 , 0  , 0  , 0  , 0  , 0  , 0  , 0  ],
    [0  ,  1 ,  0 , 0  , 0  , 0  , 0  , 0  , 0  , 0  ]])

mse(y_true, y_pred)

OUT: 0.02350000000000001

Answer 3

In general,
when we discuss this topic, it is around the idea of Stochastic GD, Mini-batch GD, Batch GD. The idea of averaging is to move the gradient towards an average of the Batch.
So, "N" refers to the batch_size in general.

On aggregation at the Output layer, it is more of a design implementation I believe.
Check this example for Keras

import tensorflow as tf, numpy as np
from tensorflow.python.keras.layers import Input, Dense
from tensorflow.python.keras.models import Model, Sequential
import keras.backend as K

model = Sequential()
model.add(Dense(3, activation="relu", input_shape=(2,), kernel_initializer=tf.keras.initializers.Ones()))
model.add(Dense(2, activation="relu", kernel_initializer=tf.keras.initializers.Ones()))

x = tf.constant([[2.,2.],[4.,4.],[5.,5.]])
y =tf.constant([[10.,12.], [20.,20.], [25.,25.]])

 # This is Keras standard definition i.e. reduce to mean on last axis
def mse_loss(y_true, y_pred): return K.mean(K.square(y_pred - y_true), axis=-1)
model.compile(loss=mse_loss) 

from keras.callbacks import LambdaCallback

def batchOutput(batch, logs): print("Finished batch: " + str(batch), end=''); print("--->",logs)
batchLogCallback = LambdaCallback(on_batch_end=batchOutput)

model.fit(x,y, batch_size=1, epochs=1,callbacks=[batchLogCallback], verbose=0, shuffle=False)

Finished batch: 0---> {'loss': 2.0}
Finished batch: 1---> {'loss': 8.509464263916016}
Finished batch: 2---> {'loss': 12.874275207519531}

You can manually calculate the loss as the input and the weights have been taken as simple integers.
The final loss [4, 0] is averaged to 2.

def mse_loss(y_true, y_pred): return K.sum(K.square(y_pred - y_true), axis=-1)

If you use the above function output will be,

Finished batch: 0---> {'loss': 4.0}
Finished batch: 1---> {'loss': 17.01892852783203}
Finished batch: 2---> {'loss': 25.748550415039062}