# What changes is the Neural Network back-propagation algorithm doing on the weights?

I have seen the formula for back-propagation algorithm for neural network error minimization, but I am not quite sure about what changes it is performing on the weights individually.

Let us suppose a simple neural network as follows:

All the weights between the connection nodes are initialized to 1.

There is no activation function, so dot product of the weights and inputs from the previous level is transmitted unchanged to the next level.

Here, the original inputs are 1 and 1. The final outputs thus becomes 4 in this case. Suppose we would like to change the weights such that the "corrected" output will exactly match the desired target output=1.

My question is, what will the resulting weight values be (as in the above diagram) after applying changes prescribed by the back-propagation algorithm?

Thank you.

• did you try to run backpropagation by hand and see for yourself? Oct 28, 2021 at 9:28
• I was about to write the same comment as @NikosM.. Furthermore, you need to specify learning rate, biases (I guess they are zero, but it is my interpretation), number of steps, etc. Oct 28, 2021 at 9:34

As already mentioned in the comments, I strongly advise doing this kind of computations (at least initially and in simple cases like this one) by hand.

In any case, here is a python code that you can play with that does what you want.

NOTE: the results dramatically depend on the learning rate and the number of epochs.

import numpy as np # I only need numpy for this

# Cost function
def J(y_true, y_pred):
"""
Cost or Loss function
"""
return ((y_true - y_pred) ** 2).mean()

class NeuralNetwork:
"""
A neural network with:
- 2 inputs
- a hidden layer with 2 neurons (a1, a2)
- an output layer with 1 neuron (h)
"""

def __init__(self):
# Initialize weights and biases
self.beta0 = np.ones(shape=(2,3))
self.beta1 = np.ones(shape=(1,3))
self.beta0[0,0] = 0
self.beta0[1,0] = 0
self.beta1[0,0] = 0

def feedforward(self, x):
a11 = self.beta0[0,1] * x[0] + self.beta0[0,2] * x[1] + self.beta0[0,0]
a12 = self.beta0[1,1] * x[0] + self.beta0[1,2] * x[1] + self.beta0[1,0]
h = self.beta1[0,1] * a11 + self.beta1[0,2] * a12 + self.beta1[0,0]
return h

def train(self, ground_truth_dataset, epoch, lr):
# ground_truth_dataset: has shape (n, 3), where n is the number of items.
# epoch: the number of times to loop through the entire ground truth dataset
# lr: learning rate

epochs = []
min_losses = []
avg_losses = []
max_losses = []
y_trues = np.array(ground_truth_dataset)[:, 2]

for ep in range(epoch):
costs = []
for item in ground_truth_dataset:
# input
x1, x2 = item[:2]

# real result
y_true = item[2]

# ====== Feed forward ======
# Neuron a1
z1 = self.beta0[0,1] * x1 + self.beta0[0,2] * x2 + self.beta0[0,0]
a1 = z1 # I use such identity her for the sake of clarity, since you do not have activation function.

# Neuron a2
z2 = self.beta0[1,1] * x1 + self.beta0[1,2] * x2 + self.beta0[1,0]
a2 = z2

# Neuron h
z = self.beta1[0,1] * a1 + self.beta1[0,2] * a2 + self.beta1[0,0]
h = z
y_pred = h

cost = J(y_true, y_pred)
costs.append(cost)

# ====== Back propagation ======
# Calculate gradients for OUTPUT layer
dJ_dy_pred = -2 * (y_true - y_pred)

dJ_db101 = dJ_dy_pred  * a1
dJ_db102 = dJ_dy_pred  * a2
dJ_db100 = dJ_dy_pred

# Calculate gradients for HIDDEN layer
dJ_db001 = dJ_dy_pred * self.beta1[0,1] * x1
dJ_db002 = dJ_dy_pred * self.beta1[0,1] * x2
dJ_db011 = dJ_dy_pred * self.beta1[0,2] * x1
dJ_db012 = dJ_dy_pred * self.beta1[0,2] * x2
dJ_db000 = dJ_dy_pred * self.beta1[0,1]
dJ_db010 = dJ_dy_pred * self.beta1[0,2]

# Update weights and biases
self.beta0[0,1] -= lr * dJ_db001
self.beta0[0,2] -= lr * dJ_db002
self.beta0[1,1] -= lr * dJ_db011
self.beta0[1,2] -= lr * dJ_db012
self.beta1[0,1] -= lr * dJ_db101
self.beta1[0,2] -= lr * dJ_db102

self.beta0[0,0] -= lr * dJ_db000
self.beta0[1,0] -= lr * dJ_db010
self.beta1[0,0] -= lr * dJ_db100

epochs.append(ep)
min_losses.append(min(costs))
avg_losses.append(sum(costs) / len(costs))
max_losses.append(max(costs))
if ep % 10 == 0:
print("Epoch {}: min_loss = {}, avg_loss = {}, max_loss = {}".format(
ep, min_losses[ep], avg_losses[ep], max_losses[ep]))

print(f"w_1 = {self.beta0[0,1]}, w_2 = {self.beta0[0,2]}, w_3 = {self.beta0[1,1]}, w_4 = {self.beta0[1,2]}, w_5 = {self.beta1[0,1]}, w_6 = {self.beta1[0,2]}")

return (epochs, min_losses, avg_losses, max_losses)



Now you are ready to train your network. You need to specify all the training parameters and give a training set:

%matplotlib inline
%config InlineBackend.figure_format = "retina"

import matplotlib.pyplot as plt
import pickle

ground_truth_dataset = [
[1, 1, 1]
]

epoch = 10
learning_rate = 0.1

n = NeuralNetwork()

stats = n.train(ground_truth_dataset, epoch, learning_rate)
epochs = stats[0]
min_losses = stats[1]
avg_losses = stats[2]
max_losses = stats[3]

plt.figure(figsize=(16,10))
plt.ylabel("Loss")
plt.xlabel("Epoch")
plt.plot(epochs, min_losses, label="Min loss")
plt.plot(epochs, avg_losses, label="Avg loss")
plt.plot(epochs, max_losses, label="Max loss")
plt.legend(loc="center")
plt.show();


Enjoy!

Additional Note: you may notice that your weights are all the same. These because you initialised them in a symmetric way, rather than in a random one. Furthermore having no activation function means having no non-linearity. This implies your 2-layers network is equivalent to a linear regressor.

• The note is an important one. Neural network wouldn't work if all weights are initialized to be the same, since they will be updated equally as well. Oct 28, 2021 at 16:32
• @James please see What should I do when someone answers my question? Oct 28, 2021 at 18:43