I want to use neural network to solve a simple regression problem, and I try to program by myself accroding to lecture Backpropagation and Neural Networks . However, I meet loss divergence problem.
My neural network can be descried as:
$l_{1}=\frac{1}{1+e^{-(W_0 x + b_0)}}$
$l_{2}={W_1 l_1 + b_1}$
And loss is:
$loss = {(y-l_2)^T(y - l_2)}$
import numpy as np
import matplotlib.pyplot as plt
np.random.seed(1)
x_pre = 2*np.random.normal(size = (1000,1))
y = x_pre**2 + -0.2*np.cos(3*np.pi*x_pre)
y = y.reshape(y.shape[0],1)
def standard(data):
mu = np.mean(data)
std = np.var(data)
return (data - mu)/std
x = standard(x_pre)
ndim = 10
w0 = 2*np.random.random((1,ndim))-1
w1 = 2*np.random.random((ndim,1))-1
np.random.seed(10)
b0 = np.random.normal(size = (1000,ndim))
b1 = np.random.normal(size = (1000,1))
lr = 1
for j in range(4):
l1 = 1/(1+ np.exp(-(np.dot(x,w0)+ b0)))
l2 = np.dot(l1,w1)+ b1#1/(1+ np.exp(-(np.dot(l1,w1)+ b1)))
l2_delta= np.mean(y - l2)*2*(y-l2)#mse
l1_delta = l2_delta.dot(w1.T)*(l1*(1-l1))
w1 += lr*l1.T.dot(l2_delta)
w0 += lr*x.T.dot(l1_delta)
b0 += lr* l1_delta
b1 += lr*l2_delta
print('loss',np.mean(y - l2))
l1 = 1/(1+ np.exp(-(np.dot(x,w0))))
l2 = 1/(1+ np.exp(-np.dot(l1,w1)))
y_hap = l2
print(np.sum(np.square(y-y_hap)))
plt.plot(np.arange(len(y)),y,'r')
plt.plot(np.arange(len(y_hap)),y_hap,'g')
plt.show()
output result:
loss 3.485080512494127
loss -30525.316587393125
loss -3293457250652.145
loss -5.777133209515429e+28
Anyone knows how to solve it?