# Bug in single layer Adaline Neuron implementation

I am trying to implement a single layer Adaline neuron, with the following mathematical foundation:

The cost function is defined as:

The weight update is defined as:

inserting the partial derivative the final weight update becomes:

My Python implementation looks as follows:

class Adaline:
n_iter = 0
eta = 0

weights_ = []

def __init__(self, eta=0.01, n_iter=50):
self.eta_ = eta
self.n_iter = n_iter

def fit(self, X, y, seed=123456):
np.random.seed(seed)
self.weights_ = np.random.normal(0, 0.1, len(X[0]) + 1)

for iter in range(self.n_iter):
weights_update = np.full(len(self.weights_), 0, dtype="float64")
cost = 0

for x, y_actual in zip(X, y):
y_predicted = np.dot(self.weights_[1:], x) + self.weights_[0]
update = (y_actual - y_predicted)

cost += update ** 2

weights_update += np.multiply(np.full(len(self.weights_), update),
np.concatenate([[1.0], x]))

cost = 0.5 * cost
weights_update *= self.eta_
self.weights_ = np.add(self.weights_, weights_update)

print("iter (" + str(iter) + ")\n J =", cost, "\n wΔ =", weights_update, "\n w =", self.weights_)

return self

def predict(self, x):
z = np.dot(self.weights_[1:], x) + self.weights_[0]
return 1 if z >= 0 else -1


I train the neuron with widely known the iris dataset:

df = pd.read_csv("iris.data", header=None, encoding='utf-8')
y = df.iloc[0:100, 4].values
y = np.where(y == 'Iris-setosa', -1, 1)
X = df.iloc[0:100, [0, 2]].values

adaline = Adaline(0.01, 3).fit(X, y)


The following output shows something doesn't work. The cost ("J") keeps on increasing all the time, as do the weights, which of course is undesired:

iter (0)
J = 89.6784555395353
wΔ = [0.53973585 3.54224292 3.27735598]
w = [0.58664708 3.51395659 3.12645013]
iter (1)
J = 42892.97766680374
wΔ = [ -28.75940385 -160.64752234  -90.03319066]
w = [ -28.17275677 -157.13356575  -86.90674053]
iter (2)
J = 66918619.409713484
wΔ = [1136.57758637 6347.70919273 3552.29203057]
w = [1108.40482961 6190.57562699 3465.38529004]


The bug seems to be in my fit function, but I can't find it.

Thanks for the help!

• By looking at your weight updates (first positive then negative) it seems that the gradient update is heading in the right direction. Have you tried decreasing your learning rate? A learning rate of 0.01 can work quite well but given that you aren't scaling your variables and aren't taking the mean for the cost function your weight update is probably too big. May 15 at 18:51
• I tried with different learning rates but the problem remains.
– Erik
May 16 at 6:50