0
$\begingroup$

I'm new in deep learning and trying to create a basic neural network for sin function. I have generated numpy array with random values of shape (50,1) which outputs sin(input).

The neural network includes one Hidden layer (3 neurons) with activation function as sigmoid and final output neuron also consists sigmoid activation function. After training the network with epoch 20000 and learning rate as 0.1 the final error does not seem to reach any close to zero.

Full Code:

import numpy as np
import pandas as pd
import random
np.random.seed(1000)
input_array=np.random.uniform(size=(50,1))
output_array=np.sin(input_array)
total_rows=input_array.shape[0]

def sigmoid(x):
    d=(1/(1+np.exp(-x)))
    return d


def derivative_sigmoid(x):
    derivative=x*(1-x)
    return derivative


epoch=20000
hidden_layer=3
input_neurons=1
output_neurons=1
learning_rate=0.1
input_array=data['input'].values.reshape(total_rows,1)
output_array=data['output'].values.reshape(total_rows,1)    
weights_in=np.random.uniform(size=(input_neurons,hidden_layer)) 
bias_in=np.random.uniform(size=(1,hidden_layer))
weights_out=np.random.uniform(size=(hidden_layer,output_neurons))
bias_out=np.random.uniform(size=(1,output_neurons))

for i in range(epoch):

    #forward propogation

    hidden_layer_output=np.dot(input_array,weights_in)+bias_in
    activation_1=sigmoid(hidden_layer_output)
    activation_2_input=np.dot(activation_1,weights_out)+bias_out
    predicted_output=sigmoid(activation_2_input)


    # #backward propogation

    Error=(predicted_output-output_array)    
    rate_change_output=derivative_sigmoid(predicted_output)
    rate_change_hidden_output=derivative_sigmoid(activation_1)
    error_on_output=Error*rate_change_output
    error_hidden_layer=error_on_output.dot(weights_out.T)
    delta_hidden_layer=error_hidden_layer*rate_change_hidden_output
    weights_out+=activation_1.T.dot(error_on_output)*learning_rate
    weights_in+=input_array.T.dot(delta_hidden_layer)*learning_rate
    bias_out+=np.sum(error_on_output,axis=0,keepdims=True)*learning_rate
    bias_in+=np.sum(error_hidden_layer,axis=0,keepdims=True)*learning_rate
print (Error)

The reason I have used sigmoid functions as the final output is in range between 0-1 (Please correct me if I'm wrong). Here is the error.

Can anyone please suggest me what's possibly I'm doing wrong?

$\endgroup$
  • $\begingroup$ Can you get the network to overfit (wit hhigher number of neurons / layers? ) . For measuring this, you will have to split data between train / test. $\endgroup$ – Shamit Verma Jan 28 at 13:25
  • $\begingroup$ Check if you can overfit as Shamit suggested. Also, try writing code in some deep learning library (TF/Keras, Pytorch) so that you have gradients calculated automatically to remove the doubt of your gradient formulas being wrong. Also, since you are doing the regression problem, try using other activation function than sigmoid (sigmoids are good for classification problem). $\endgroup$ – Antonio Jurić Feb 4 at 13:01
  • $\begingroup$ Also, use mean squared error. Currently you only subtract the prediction and groundtrut and that can cause problems when you have two diffs like -5 and 5. When you some those two, you would get 0 loss instead of 10 (if you are using mean squared error, you would get 25 loss). $\endgroup$ – Antonio Jurić Feb 4 at 13:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.