# How can I optimise/parallelize my neural network code?

I have a neural network with 784 inputs, 30 hidden neurons and 10 output neurons. The main performance issue is when backpropagating. Currently it takes around 0.1 seconds for one iteration of backpropagating. As my training set is 60,000 examples, this will take a long time for me to even confirm that my algorithm is working.

I atttempted to parallelize the hidden_to_output and input_to_hidden functions, to no avail. I apologize for the messy code, but I have attached the two functions that are used for calculating the error/backpropagation.

Just wondering how I can speed this up/parallelize it without involving tensorflow.

def hidden_to_output(weights2):
for i in range(len(weights2)):
sub_list = []
for j in range(len(weights2[i])):
input = hidden[j]
w = weights2[i][j]
out = output[i]
desired_output = desired_outputs[i]
Etotal_out = -(desired_output-out)
out_net = out*(1-out)
net_w5 = hidden[j]
Etotal_w5 = Etotal_out*out_net*net_w5
w_prime = w - learning_rate*Etotal_w5
sub_list.append(w_prime)
new_weights2.append(sub_list)
return new_weights2

def input_to_hidden(weights1):
for i in range(len(weights1)):
sub_list = []
for j in range(len(weights1[i])):
E_o1_out_o1 = -(desired_outputs - output)
out_o1_net_o1 = output*(1-output)
E_o1_net_o1 = E_o1_out_o1*out_o1_net_o1
E_o1_out_h1 = E_o1_net_o1*weights2[i]

E_oN_out_o1 = -(desired_outputs - output)
out_o2_net_o2 = output*(1-output)
E_o2_net_o2 = E_oN_out_o1* out_o2_net_o2
E_o2_out_h1 = E_o2_net_o2*weights2[i]

E_total_out_h1 = E_o2_out_h1 + E_o1_out_h1
out_h1_net_h1 = hidden[i]*(1-hidden[i])
net_h1_w1 = inputs[j]
E_total_w1 = E_total_out_h1*out_h1_net_h1*net_h1_w1

w_prime = weights1[i][j] - learning_rate*E_total_w1
sub_list.append(w_prime)
new_weights1.append(sub_list)
return new_weights1

if __name__ == '__main__':

for n in range(60):
# Update weights from hidden layer to output layer.
new_weights2 = hidden_to_output(weights2)

# Update weights from input to hidden layer.
new_weights1 = input_to_hidden(weights1)

weights1 = new_weights1
weights2 = new_weights2
new_weights1 = []
new_weights2 = []
hidden = np.dot(weights1, inputs) + b1
squash(hidden)
output = np.dot(weights2, hidden) + b2
squash(output)

result = calc_total_error(output, desired_outputs)
print(f"{result:.8f}")
output = output.tolist()
print(output)
print(output.index(max(output)))
print("--- %s seconds ---" % (time.time() - start_time))

• I hope you are using numpy array wherever possible and using its speedup features like vectorization. Also, the frameworks like Tensorflow and PyTorch would be doing a lot of optimizations internally like including h/w level optimizations. And as these frameworks are opensourced, you can also look in their code. May 30, 2021 at 6:54
• @DevashishPrasad Thanks for your comment. I will look into vectorization and converting my arrays to numpy arrays. I am trying to avoid other frameworks at the moment so I can get the basics down pat before moving on. May 30, 2021 at 7:34

As mentioned by Devashish, make use of numpy arrays instead of python lists as they allow for vectorized computations and are much faster. Instead of interating over the python lists and multiplying values you can simply use vectorized functions such as numpy.dot to multiply the weight matrices with the input.