Logistic Regression Manual Update

Question

For the logistic regression below, how can I manually update the coefficients a and b manually? EDIT

y = 1.0 / (1.0 + exp(-ax - b))

after observing the following data examples (x, y), the parameters a and b need to be updated using gradient descent with a learning rate = 1.

1. input: x = 1, y = 1, a = 1, b = 1

expected output: a = 0.881, b = 0.881

2. input: x = 2.2, y = 0, a = 5.1, b = 5.7

expected output: a = 7.3, b = 6.7

UPDATE

def update(x, y, a, b):
    t = 1 + np.exp(-a * x - b)
    da = (y - 1 / t) * x * (t - 1) * t**-2
    db = (y - 1 / t) * (t - 1) * t**-2
    new_a = a - da
    new_b = b - db
    return new_a, new_b

This got me

update(1, 1, 1, 1)
(0.9874844578263232, 0.9874844578263232)

Did I miss something here?

FINAL

def manual_update(x, y, a, b):
  p = 1 / (1 + np.exp(-a*x-b))
  pdpda = x*np.exp(-a*x-b) / (1+np.exp(-a*x-b))**2
  pdpdb = np.exp(-a*x-b) / (1+np.exp(-a*x-b))**2
  pdLda = -y*pdpda/p+(1-y)*pdpda/(1-p)
  pdLdb = -y*pdpdb/p+(1-y)*pdpdb/(1-p)
  return round(a + pdLda, 3), round(b + pdLdb, 3)

this worked perfectly.

Answer 1

I believe the answer given by Anoop A Nair is wrong. It takes me few time to recognise that I fall in same trap like Anoop A Nair.

The question said it is Logistic Regression. So the loss function should be cross entropy. Shouldn't be Squared Error loss.

Answer 2

The update rules are ;

$a := a - \alpha.\frac{\partial L}{\partial a}$

$b := b - \alpha.\frac{\partial L}{\partial b}$

Here L which is the Loss [Squared Error in this case ] is given by;

$L = \frac{1}{2}.(\hat{y} - \frac{1.0}{(1.0 + exp^{-ax + b})})^{2}$

So ;

$\frac{\partial L}{\partial a} = \frac{1}{2}.2(\hat{y} - \frac{1.0}{(1.0 + exp^{-ax + b})}).\frac{-1.0}{(1.0 + exp^{-ax + b})^{2}}.(-x).e^{-ax+b} $

$\frac{\partial L}{\partial b} = \frac{1}{2}.2(\hat{y} - \frac{1.0}{(1.0 + exp^{-ax + b})}).\frac{-1.0}{(1.0 + exp^{-ax + b})^{2}}.e^{-ax+b} $

So now guess an intial value for a and b. You can plug in the an x and y pair for and update the weights simultaneously. The thing to be careful is that when updating for a and b in an iteration use the previous a and b values. That is if you have updated your a prior to b dont use the updated value of a to update b.