0
$\begingroup$

I am trying to implement a single layer Adaline neuron, with the following mathematical foundation:

The cost function is defined as: enter image description here

The weight update is defined as:

enter image description here

enter image description here

inserting the partial derivative the final weight update becomes:

enter image description here

My Python implementation looks as follows:

class Adaline:
    n_iter = 0
    eta = 0

    weights_ = []

    def __init__(self, eta=0.01, n_iter=50):
        self.eta_ = eta
        self.n_iter = n_iter

    def fit(self, X, y, seed=123456):
        np.random.seed(seed)
        self.weights_ = np.random.normal(0, 0.1, len(X[0]) + 1)

        for iter in range(self.n_iter):
            weights_update = np.full(len(self.weights_), 0, dtype="float64")
            cost = 0

            for x, y_actual in zip(X, y):
                y_predicted = np.dot(self.weights_[1:], x) + self.weights_[0]
                update = (y_actual - y_predicted)

                cost += update ** 2

                weights_update += np.multiply(np.full(len(self.weights_), update),
                                              np.concatenate([[1.0], x]))


            cost = 0.5 * cost
            weights_update *= self.eta_
            self.weights_ = np.add(self.weights_, weights_update)

            print("iter (" + str(iter) + ")\n J =", cost, "\n wΔ =", weights_update, "\n w =", self.weights_)

        return self

    def predict(self, x):
        z = np.dot(self.weights_[1:], x) + self.weights_[0]
        return 1 if z >= 0 else -1

I train the neuron with widely known the iris dataset:

df = pd.read_csv("iris.data", header=None, encoding='utf-8')
y = df.iloc[0:100, 4].values
y = np.where(y == 'Iris-setosa', -1, 1)
X = df.iloc[0:100, [0, 2]].values

adaline = Adaline(0.01, 3).fit(X, y)

The following output shows something doesn't work. The cost ("J") keeps on increasing all the time, as do the weights, which of course is undesired:

iter (0)
 J = 89.6784555395353 
 wΔ = [0.53973585 3.54224292 3.27735598] 
 w = [0.58664708 3.51395659 3.12645013]
iter (1)
 J = 42892.97766680374 
 wΔ = [ -28.75940385 -160.64752234  -90.03319066] 
 w = [ -28.17275677 -157.13356575  -86.90674053]
iter (2)
 J = 66918619.409713484 
 wΔ = [1136.57758637 6347.70919273 3552.29203057] 
 w = [1108.40482961 6190.57562699 3465.38529004]

The bug seems to be in my fit function, but I can't find it.

Thanks for the help!

$\endgroup$
2
  • $\begingroup$ By looking at your weight updates (first positive then negative) it seems that the gradient update is heading in the right direction. Have you tried decreasing your learning rate? A learning rate of 0.01 can work quite well but given that you aren't scaling your variables and aren't taking the mean for the cost function your weight update is probably too big. $\endgroup$ – Oxbowerce May 15 at 18:51
  • $\begingroup$ I tried with different learning rates but the problem remains. $\endgroup$ – Erik May 16 at 6:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.