I am applying the simple least mean square update rule using python but somehow the values of theta, I get, become very high.

from pylab import * 
data = array( 

x = zeros( (len(data[:,4]) ,2)) 
x[:,0] ,x[:,1] = 1, data[:,4] 
y  = data[:,-1] 
theta = array([100.0,100.0]) 
alpha  = 0.4 
iternum = 100 
for i in range(iternum):    
    theta -= alpha*dot(transpose(x),(dot(x,theta)-y)) 
print theta

I get the answer to be [7.18957001e+150 1.19047264e+151] which is unrealistic for the given code.

However if I alter the internum loop to be

 for i in range(iternum):    
      theta -= alpha*dot(transpose(x),(dot(x,theta)-y))/size(data[:,4])  #Basically divide by the total number of training examples
 print theta

I get the correct answer. However, as per what I have learned, the cost function does not necessarily depend on training example size.

Can somebody point to the source of the problem?

Apologies if the explanation of the problem was a little convoluted.


1 Answer 1


Your second change (calculating the average error) is the correct method. Imagine this: if there are billion examples in your training set, even if you make (very) small error on each, the sum total is going to add up to a large number. So the cost function in case of least squared is mean squared error, not just sum.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.