# My ADALINE model using Gradient Descent is increasing error on each iteration

I have used the Iris Dataset's 1st and 3rd Column for the features. and the labels of Iris Setosa (-1) and Iris Versicolor (1). I am using ADALINE as a simple classification model for my dataset. I am using gradient descent as the cost minimizing function. But on every iteration the error increases. What am I doing wrong in the python code?

import numpy as np
import pandas as pd

def __init__(self, eta = 0.01, n_iter = 50):
self.eta = eta
self.n_iter = n_iter

def fit (self, X, y):
"""Fit training data."""

self.w_ = np.random.random(X.shape[1])
self.cost_ = []
print ('Initial weights are: %r' %self.w_)
for i in range(self.n_iter):
output = self.net_input(X)
print ("On iteration %d, output is: %r" %(i, output))
errors = output - y
print("On iteration %d, Error is: %r" %(i, errors))
self.w_ += self.eta * X.T.dot(errors)
print ('Weights on iteration %d: %r' %(i, self.w_))
cost = (errors**2).sum() / 2.0
self.cost_.append(cost)
print ("On iteration %d, Cost is: %r" %(i, cost))
prediction = self.predict(X)
print ("Prediction after iteration %d is: %r" %(i, prediction))
input()
return self

def net_input(self, X):
"""Calculate net input"""
return X.dot(self.w_)

def activation(self, X):
"""Computer Linear Activation"""
return self.net_input(X)

def predict(self, X):
"""Return class label after unit step"""
return np.where(self.activation(X) >= 0.0, 1, -1)

####### END OF THE CLASS ########
#importing the Iris Dataset
y = df.iloc[0:100, 4].values
y = np.where(y == 'Iris-setosa', -1, 1)
X = df.iloc[0:100, [0, 2]].values
#Adding the ones column to the X matrix
X = np.insert(X, 0,  np.ones(X.shape[0]), axis = 1)


I think something is wrong here.

self.w_ += self.eta * X.T.dot(errors)


You are going to the positive to the gradient while you should be doing is going to the negative direction of it

self.w_ -= self.eta * X.T.dot(errors)


or

self.w_ += -self.eta * X.T.dot(errors)


see this for more clarification.

• After making the direction negative on the gradient descent, it only started working when I decreased the learning rate to 0.0001 from 0.001. On 0.001 it kept on switching predictions from 1 to -1 on each iteration. Jan 15 '17 at 16:24
• Indeed, you have to choose carefully your learning rate. If it is too big, your algorithm diverge. There are different ways to find a learning rate adapted to your situation, maybe this paper (part 5.1) will help you: cs.cmu.edu/~ggordon/10725-F12/scribes/10725_Lecture5.pdf Jan 15 '17 at 17:59
• Adding to @Pierre 's comment, take an sample function say $x^2+4$ and start with a guess say $5$ and keep changing the learning rates from $1$ to $0.1$ to $0.01$. You can the values of future $x$ being just jumping around the minimum in one case and lowering the learning rate stops this. But a learning rate above than this can sometimes do the same job of convergence more quicker as in the case of $0.1$ to $0.01$. Jan 15 '17 at 18:38

If you want to do

self.w_ += self.eta * X.T.dot(errors)


like i like to do.

you just have to change

errors = output - y


to

errors = y - output


Hope this helps : )