0
$\begingroup$

I have a neural network with 784 inputs, 30 hidden neurons and 10 output neurons. The main performance issue is when backpropagating. Currently it takes around 0.1 seconds for one iteration of backpropagating. As my training set is 60,000 examples, this will take a long time for me to even confirm that my algorithm is working.

I atttempted to parallelize the hidden_to_output and input_to_hidden functions, to no avail. I apologize for the messy code, but I have attached the two functions that are used for calculating the error/backpropagation.

Just wondering how I can speed this up/parallelize it without involving tensorflow.

def hidden_to_output(weights2):
for i in range(len(weights2)):
    sub_list = []
    for j in range(len(weights2[i])):
        input = hidden[j]
        w = weights2[i][j]
        out = output[i]
        desired_output = desired_outputs[i]
        Etotal_out = -(desired_output-out)
        out_net = out*(1-out)
        net_w5 = hidden[j]
        Etotal_w5 = Etotal_out*out_net*net_w5
        w_prime = w - learning_rate*Etotal_w5
        sub_list.append(w_prime)
    new_weights2.append(sub_list) 
return new_weights2

def input_to_hidden(weights1):
    for i in range(len(weights1)):
        sub_list = []
        for j in range(len(weights1[i])):
            E_o1_out_o1 = -(desired_outputs[0] - output[0])
            out_o1_net_o1 = output[0]*(1-output[0])
            E_o1_net_o1 = E_o1_out_o1*out_o1_net_o1
            E_o1_out_h1 = E_o1_net_o1*weights2[0][i]

            E_oN_out_o1 = -(desired_outputs[1] - output[1])
            out_o2_net_o2 = output[1]*(1-output[1])
            E_o2_net_o2 = E_oN_out_o1* out_o2_net_o2
            E_o2_out_h1 = E_o2_net_o2*weights2[1][i]
            
            E_total_out_h1 = E_o2_out_h1 + E_o1_out_h1
            out_h1_net_h1 = hidden[i]*(1-hidden[i])
            net_h1_w1 = inputs[j]
            E_total_w1 = E_total_out_h1*out_h1_net_h1*net_h1_w1

            w_prime = weights1[i][j] - learning_rate*E_total_w1
            sub_list.append(w_prime)
        new_weights1.append(sub_list)
    return new_weights1

if __name__ == '__main__':

    for n in range(60):
        # Update weights from hidden layer to output layer.
        new_weights2 = hidden_to_output(weights2)

        # Update weights from input to hidden layer.
        new_weights1 = input_to_hidden(weights1)

        weights1 = new_weights1
        weights2 = new_weights2
        new_weights1 = []
        new_weights2 = []
        hidden = np.dot(weights1, inputs) + b1
        squash(hidden)
        output = np.dot(weights2, hidden) + b2
        squash(output)

    result = calc_total_error(output, desired_outputs)
    print(f"{result:.8f}")
    output = output.tolist()
    print(output)
    print(output.index(max(output)))
    print("--- %s seconds ---" % (time.time() - start_time))
Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ I hope you are using numpy array wherever possible and using its speedup features like vectorization. Also, the frameworks like Tensorflow and PyTorch would be doing a lot of optimizations internally like including h/w level optimizations. And as these frameworks are opensourced, you can also look in their code. $\endgroup$
    – Devashish Prasad
    Commented May 30, 2021 at 6:54
  • $\begingroup$ @DevashishPrasad Thanks for your comment. I will look into vectorization and converting my arrays to numpy arrays. I am trying to avoid other frameworks at the moment so I can get the basics down pat before moving on. $\endgroup$
    – Tiltlord
    Commented May 30, 2021 at 7:34

1 Answer 1

Reset to default
0
$\begingroup$

As mentioned by Devashish, make use of numpy arrays instead of python lists as they allow for vectorized computations and are much faster. Instead of interating over the python lists and multiplying values you can simply use vectorized functions such as numpy.dot to multiply the weight matrices with the input.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.