Why the loss is nan by using linear activation function in the last layer?

Question

I want to use neural network to solve a simple regression problem, and I try to program by myself accroding to lecture Backpropagation and Neural Networks . However, I meet loss divergence problem.

My neural network can be descried as:

$l_{1}=\frac{1}{1+e^{-(W_0 x + b_0)}}$

$l_{2}={W_1 l_1 + b_1}$

And loss is:

$loss = {(y-l_2)^T(y - l_2)}$

import numpy as np
import matplotlib.pyplot as plt

np.random.seed(1)
x_pre = 2*np.random.normal(size = (1000,1))
y = x_pre**2 + -0.2*np.cos(3*np.pi*x_pre) 
y = y.reshape(y.shape[0],1)

def standard(data):
    mu = np.mean(data)
    std = np.var(data)
    return (data - mu)/std
x = standard(x_pre)

ndim = 10
w0 = 2*np.random.random((1,ndim))-1
w1 = 2*np.random.random((ndim,1))-1
np.random.seed(10)
b0 = np.random.normal(size = (1000,ndim))
b1 = np.random.normal(size = (1000,1))
lr = 1
for j in range(4):
    l1 = 1/(1+ np.exp(-(np.dot(x,w0)+ b0))) 
    l2 = np.dot(l1,w1)+ b1#1/(1+ np.exp(-(np.dot(l1,w1)+ b1)))
    l2_delta= np.mean(y - l2)*2*(y-l2)#mse
    l1_delta = l2_delta.dot(w1.T)*(l1*(1-l1))
    w1 += lr*l1.T.dot(l2_delta)
    w0 += lr*x.T.dot(l1_delta)
    b0 += lr* l1_delta
    b1 += lr*l2_delta
    print('loss',np.mean(y - l2))
l1 = 1/(1+ np.exp(-(np.dot(x,w0))))
l2 = 1/(1+ np.exp(-np.dot(l1,w1)))
y_hap =  l2

print(np.sum(np.square(y-y_hap)))
plt.plot(np.arange(len(y)),y,'r')
plt.plot(np.arange(len(y_hap)),y_hap,'g')
plt.show()

output result:

loss 3.485080512494127
loss -30525.316587393125
loss -3293457250652.145
loss -5.777133209515429e+28

Anyone knows how to solve it?

Answer 1

There seem a lot of initialization problems.

1. There ain't too many biases. Its just 1 per layer. Your shape of biases seems wrong.

2. You may have got the network right in shapes since it doesn't return shape errors. But bear in mind, it seems highly likely #1 is true.

3. Same goes to w0 and w1.

Consider checking all dims again. If you need, I'm attaching a sample Neural Network implementation in python.

https://raw.githubusercontent.com/MasterSkepticista/NEAT_py/master/network.py

This is as it happens to be taught in Andrew Ng's course.