3
$\begingroup$

I have seen the formula for back-propagation algorithm for neural network error minimization, but I am not quite sure about what changes it is performing on the weights individually.

Let us suppose a simple neural network as follows:

enter image description here

All the weights between the connection nodes are initialized to 1.

There is no activation function, so dot product of the weights and inputs from the previous level is transmitted unchanged to the next level.

Here, the original inputs are 1 and 1. The final outputs thus becomes 4 in this case. Suppose we would like to change the weights such that the "corrected" output will exactly match the desired target output=1.

enter image description here

My question is, what will the resulting weight values be (as in the above diagram) after applying changes prescribed by the back-propagation algorithm?

Thank you.

$\endgroup$
2
  • 1
    $\begingroup$ did you try to run backpropagation by hand and see for yourself? $\endgroup$
    – Nikos M.
    Oct 28, 2021 at 9:28
  • $\begingroup$ I was about to write the same comment as @NikosM.. Furthermore, you need to specify learning rate, biases (I guess they are zero, but it is my interpretation), number of steps, etc. $\endgroup$
    – Oscar
    Oct 28, 2021 at 9:34

1 Answer 1

2
$\begingroup$

As already mentioned in the comments, I strongly advise doing this kind of computations (at least initially and in simple cases like this one) by hand.

In any case, here is a python code that you can play with that does what you want.

NOTE: the results dramatically depend on the learning rate and the number of epochs.

import numpy as np # I only need numpy for this

# Cost function 
def J(y_true, y_pred):
    """
        Cost or Loss function
    """ 
    return ((y_true - y_pred) ** 2).mean()

class NeuralNetwork:
    """
        A neural network with:
            - 2 inputs
            - a hidden layer with 2 neurons (a1, a2)
            - an output layer with 1 neuron (h)
    """

    def __init__(self):
        # Initialize weights and biases
        self.beta0 = np.ones(shape=(2,3))
        self.beta1 = np.ones(shape=(1,3))
        self.beta0[0,0] = 0
        self.beta0[1,0] = 0
        self.beta1[0,0] = 0
    

    def feedforward(self, x):
        a11 = self.beta0[0,1] * x[0] + self.beta0[0,2] * x[1] + self.beta0[0,0]
        a12 = self.beta0[1,1] * x[0] + self.beta0[1,2] * x[1] + self.beta0[1,0]
        h = self.beta1[0,1] * a11 + self.beta1[0,2] * a12 + self.beta1[0,0]
        return h

    def train(self, ground_truth_dataset, epoch, lr):
        # ground_truth_dataset: has shape (n, 3), where n is the number of items.
        # epoch: the number of times to loop through the entire ground truth dataset
        # lr: learning rate

        epochs = []
        min_losses = []
        avg_losses = []
        max_losses = []
        y_trues = np.array(ground_truth_dataset)[:, 2]

        for ep in range(epoch):
            costs = []
            for item in ground_truth_dataset:
                # input
                x1, x2 = item[:2]

                # real result
                y_true = item[2]

                # ====== Feed forward ======
                # Neuron a1
                z1 = self.beta0[0,1] * x1 + self.beta0[0,2] * x2 + self.beta0[0,0]
                a1 = z1 # I use such identity her for the sake of clarity, since you do not have activation function.

                # Neuron a2
                z2 = self.beta0[1,1] * x1 + self.beta0[1,2] * x2 + self.beta0[1,0]
                a2 = z2

                # Neuron h
                z = self.beta1[0,1] * a1 + self.beta1[0,2] * a2 + self.beta1[0,0]
                h = z
                y_pred = h

                cost = J(y_true, y_pred)
                costs.append(cost)

                # ====== Back propagation ======
                # Calculate gradients for OUTPUT layer
                dJ_dy_pred = -2 * (y_true - y_pred)

                dJ_db101 = dJ_dy_pred  * a1
                dJ_db102 = dJ_dy_pred  * a2
                dJ_db100 = dJ_dy_pred

                # Calculate gradients for HIDDEN layer
                dJ_db001 = dJ_dy_pred * self.beta1[0,1] * x1
                dJ_db002 = dJ_dy_pred * self.beta1[0,1] * x2
                dJ_db011 = dJ_dy_pred * self.beta1[0,2] * x1
                dJ_db012 = dJ_dy_pred * self.beta1[0,2] * x2
                dJ_db000 = dJ_dy_pred * self.beta1[0,1]
                dJ_db010 = dJ_dy_pred * self.beta1[0,2]

                # Update weights and biases
                self.beta0[0,1] -= lr * dJ_db001
                self.beta0[0,2] -= lr * dJ_db002
                self.beta0[1,1] -= lr * dJ_db011
                self.beta0[1,2] -= lr * dJ_db012
                self.beta1[0,1] -= lr * dJ_db101
                self.beta1[0,2] -= lr * dJ_db102

                self.beta0[0,0] -= lr * dJ_db000
                self.beta0[1,0] -= lr * dJ_db010
                self.beta1[0,0] -= lr * dJ_db100

            epochs.append(ep)
            min_losses.append(min(costs))
            avg_losses.append(sum(costs) / len(costs))
            max_losses.append(max(costs))
            if ep % 10 == 0:
                print("Epoch {}: min_loss = {}, avg_loss = {}, max_loss = {}".format(
                    ep, min_losses[ep], avg_losses[ep], max_losses[ep]))

                print(f"w_1 = {self.beta0[0,1]}, w_2 = {self.beta0[0,2]}, w_3 = {self.beta0[1,1]}, w_4 = {self.beta0[1,2]}, w_5 = {self.beta1[0,1]}, w_6 = {self.beta1[0,2]}")

        return (epochs, min_losses, avg_losses, max_losses)


Now you are ready to train your network. You need to specify all the training parameters and give a training set:

%matplotlib inline
%config InlineBackend.figure_format = "retina"

import matplotlib.pyplot as plt
import pickle


ground_truth_dataset = [
  [1, 1, 1]
]

epoch = 10
learning_rate = 0.1

n = NeuralNetwork()

stats = n.train(ground_truth_dataset, epoch, learning_rate)
epochs = stats[0]
min_losses = stats[1]
avg_losses = stats[2]
max_losses = stats[3]


plt.figure(figsize=(16,10))
plt.ylabel("Loss")
plt.xlabel("Epoch")
plt.plot(epochs, min_losses, label="Min loss")
plt.plot(epochs, avg_losses, label="Avg loss")
plt.plot(epochs, max_losses, label="Max loss")
plt.legend(loc="center")
plt.show();

Enjoy!

Additional Note: you may notice that your weights are all the same. These because you initialised them in a symmetric way, rather than in a random one. Furthermore having no activation function means having no non-linearity. This implies your 2-layers network is equivalent to a linear regressor.

$\endgroup$
3
  • $\begingroup$ @ Oscar thank you very much! $\endgroup$
    – user127052
    Oct 28, 2021 at 14:17
  • 1
    $\begingroup$ The note is an important one. Neural network wouldn't work if all weights are initialized to be the same, since they will be updated equally as well. $\endgroup$
    – justhalf
    Oct 28, 2021 at 16:32
  • $\begingroup$ @James please see What should I do when someone answers my question? $\endgroup$
    – desertnaut
    Oct 28, 2021 at 18:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.