I have used the Iris Dataset's 1st and 3rd Column for the features. and the labels of Iris Setosa (-1) and Iris Versicolor (1). I am using ADALINE as a simple classification model for my dataset. I am using gradient descent as the cost minimizing function. But on every iteration the error increases. What am I doing wrong in the python code?

import numpy as np
import pandas as pd

class AdalineGD(object):

    def __init__(self, eta = 0.01, n_iter = 50):
        self.eta = eta
        self.n_iter = n_iter

    def fit (self, X, y):
        """Fit training data."""

        self.w_ = np.random.random(X.shape[1])
        self.cost_ = []
        print ('Initial weights are: %r' %self.w_)
        for i in range(self.n_iter):
            output = self.net_input(X)
            print ("On iteration %d, output is: %r" %(i, output))
            errors = output - y
            print("On iteration %d, Error is: %r" %(i, errors))
            self.w_ += self.eta * X.T.dot(errors)
            print ('Weights on iteration %d: %r' %(i, self.w_))
            cost = (errors**2).sum() / 2.0
            print ("On iteration %d, Cost is: %r" %(i, cost))
            prediction = self.predict(X)
            print ("Prediction after iteration %d is: %r" %(i, prediction))
        return self

    def net_input(self, X):
        """Calculate net input"""
        return X.dot(self.w_)

    def activation(self, X):
        """Computer Linear Activation"""
        return self.net_input(X)

    def predict(self, X):
        """Return class label after unit step"""
        return np.where(self.activation(X) >= 0.0, 1, -1)

####### END OF THE CLASS ########
#importing the Iris Dataset 
df = pd.read_csv("https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data", header = None)
y = df.iloc[0:100, 4].values
y = np.where(y == 'Iris-setosa', -1, 1)
X = df.iloc[0:100, [0, 2]].values
#Adding the ones column to the X matrix
X = np.insert(X, 0,  np.ones(X.shape[0]), axis = 1)
ada = AdalineGD(n_iter = 20, eta = 0.001).fit(X, y)

I think something is wrong here.

self.w_ += self.eta * X.T.dot(errors)

You are going to the positive to the gradient while you should be doing is going to the negative direction of it

self.w_ -= self.eta * X.T.dot(errors)


self.w_ += -self.eta * X.T.dot(errors)

see this for more clarification.

  • $\begingroup$ After making the direction negative on the gradient descent, it only started working when I decreased the learning rate to 0.0001 from 0.001. On 0.001 it kept on switching predictions from 1 to -1 on each iteration. $\endgroup$ Jan 15 '17 at 16:24
  • 2
    $\begingroup$ Indeed, you have to choose carefully your learning rate. If it is too big, your algorithm diverge. There are different ways to find a learning rate adapted to your situation, maybe this paper (part 5.1) will help you: cs.cmu.edu/~ggordon/10725-F12/scribes/10725_Lecture5.pdf $\endgroup$
    – Pierre
    Jan 15 '17 at 17:59
  • $\begingroup$ Adding to @Pierre 's comment, take an sample function say $x^2+4$ and start with a guess say $5$ and keep changing the learning rates from $1$ to $0.1$ to $0.01$. You can the values of future $x$ being just jumping around the minimum in one case and lowering the learning rate stops this. But a learning rate above than this can sometimes do the same job of convergence more quicker as in the case of $0.1$ to $0.01$. $\endgroup$ Jan 15 '17 at 18:38

If you want to do

self.w_ += self.eta * X.T.dot(errors)

like i like to do.

you just have to change

errors = output - y


errors = y - output

Hope this helps : )


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.