Logistic Regression Manual Update

For the logistic regression below, how can I manually update the coefficients a and b manually? EDIT

y = 1.0 / (1.0 + exp(-ax - b))


after observing the following data examples (x, y), the parameters a and b need to be updated using gradient descent with a learning rate = 1.

1. input: x = 1, y = 1, a = 1, b = 1

expected output: a = 0.881, b = 0.881

2. input: x = 2.2, y = 0, a = 5.1, b = 5.7

expected output: a = 7.3, b = 6.7

UPDATE

def update(x, y, a, b):
t = 1 + np.exp(-a * x - b)
da = (y - 1 / t) * x * (t - 1) * t**-2
db = (y - 1 / t) * (t - 1) * t**-2
new_a = a - da
new_b = b - db
return new_a, new_b


This got me

update(1, 1, 1, 1)
(0.9874844578263232, 0.9874844578263232)


Did I miss something here?

FINAL

def manual_update(x, y, a, b):
p = 1 / (1 + np.exp(-a*x-b))
pdpda = x*np.exp(-a*x-b) / (1+np.exp(-a*x-b))**2
pdpdb = np.exp(-a*x-b) / (1+np.exp(-a*x-b))**2
pdLda = -y*pdpda/p+(1-y)*pdpda/(1-p)
pdLdb = -y*pdpdb/p+(1-y)*pdpdb/(1-p)
return round(a + pdLda, 3), round(b + pdLdb, 3)


this worked perfectly.

The update rules are ;

$$a := a - \alpha.\frac{\partial L}{\partial a}$$

$$b := b - \alpha.\frac{\partial L}{\partial b}$$

Here L which is the Loss [Squared Error in this case ] is given by;

$$L = \frac{1}{2}.(\hat{y} - \frac{1.0}{(1.0 + exp^{-ax + b})})^{2}$$

So ;

$$\frac{\partial L}{\partial a} = \frac{1}{2}.2(\hat{y} - \frac{1.0}{(1.0 + exp^{-ax + b})}).\frac{-1.0}{(1.0 + exp^{-ax + b})^{2}}.(-x).e^{-ax+b}$$

$$\frac{\partial L}{\partial b} = \frac{1}{2}.2(\hat{y} - \frac{1.0}{(1.0 + exp^{-ax + b})}).\frac{-1.0}{(1.0 + exp^{-ax + b})^{2}}.e^{-ax+b}$$

So now guess an intial value for a and b. You can plug in the an x and y pair for and update the weights simultaneously. The thing to be careful is that when updating for a and b in an iteration use the previous a and b values. That is if you have updated your a prior to b dont use the updated value of a to update b.

I believe the answer given by Anoop A Nair is wrong. It takes me few time to recognise that I fall in same trap like Anoop A Nair.

The question said it is Logistic Regression. So the loss function should be cross entropy. Shouldn't be Squared Error loss.

• so are you saying that the loss functio is L = y^ - 1 / (1 - exp(-ax-b))? – Hiro Nakagame Jan 19 at 20:12
• I don't be sure about this. But I saw Andrew Ng's course, it say it should be: Loss = −(ylog(p)+(1−y)log(1−p)) – NoahShen Jan 20 at 2:44
• yes it worked! thanks very much. For anyone who is interested, there is the link to Andrew's video. youtube.com/watch?v=TTdcc21Ko9A – Hiro Nakagame Jan 20 at 5:00