I am applying the simple least mean square update rule using python but somehow the values of theta, I get, become very high.
from pylab import * data = array( [[1,4.9176,1.0,3.4720,0.998,1.0,7,4,42,3,1,0,25.9], [2,5.0208,1.0,3.5310,1.50,2.0,7,4,62,1,1,0,29.5], [3,4.5429,1.0,2.2750,1.175,1.0,6,3,40,2,1,0,27.9], [4,4.5573,1.0,4.050,1.232,1.0,6,3,54,4,1,0,25.9], [5,5.0597,1.0,4.4550,1.121,1.0,6,3,42,3,1,0,29.9], [6,3.8910,1.0,4.4550,0.988,1.0,6,3,56,2,1,0,29.9], [7,5.8980,1.0,5.850,1.240,1.0,7,3,51,2,1,1,30.9], [8,5.6039,1.0,9.520,1.501,0.0,6,3,32,1,1,0,28.9], [9,16.4202,2.5,9.80,3.420,2.0,10,5,42,2,1,1,84.9], [10,14.4598,2.5,12.80,3.0,2.0,9,5,14,4,1,1,82.9], [11,5.8282,1.0,6.4350,1.225,2.0,6,3,32,1,1,0,35.9], [12,5.303,1.0,4.9883,1.552,1.0,6,3,30,1,2,0,31.5], [13,6.2712,1.0,5.520,0.975,1.0,5,2,30,1,2,0,31.0], [14,5.9592,1.0,6.6660,1.121,2.0,6,3,32,2,1,0,30.9], [15,5.050,1.0,5.0,1.020,0.0,5,2,46,4,1,1,30.0], [16,5.6039,1.0,9.520,1.501,0.0,6,3,32,1,1,0,28.9], [17,8.2464,1.5,5.150,1.664,2.0,8,4,50,4,1,0,36.9], [18,6.6969,1.5,6.9020,1.488,1.5,7,3,22,1,1,1,41.9], [19,7.7841,1.5,7.1020,1.376,1.0,6,3,17,2,1,0,40.5], [20,9.0384,1.0,7.80,1.50,1.5,7,3,23,3,3,0,43.9], [21,5.9894,1.0,5.520,1.256,2.0,6,3,40,4,1,1,37.5], [22,7.5422,1.5,4.0,1.690,1.0,6,3,22,1,1,0,37.9], [23,8.7951,1.5,9.890,1.820,2.0,8,4,50,1,1,1,44.5]]) x = zeros( (len(data[:,4]) ,2)) x[:,0] ,x[:,1] = 1, data[:,4] y = data[:,-1] theta = array([100.0,100.0]) alpha = 0.4 iternum = 100 for i in range(iternum): theta -= alpha*dot(transpose(x),(dot(x,theta)-y)) print theta
I get the answer to be [7.18957001e+150 1.19047264e+151] which is unrealistic for the given code.
However if I alter the internum loop to be
for i in range(iternum): theta -= alpha*dot(transpose(x),(dot(x,theta)-y))/size(data[:,4]) #Basically divide by the total number of training examples print theta
I get the correct answer. However, as per what I have learned, the cost function does not necessarily depend on training example size.
Can somebody point to the source of the problem?
Apologies if the explanation of the problem was a little convoluted.