0
$\begingroup$

For the logistic regression below, how can I manually update the coefficients a and b manually? EDIT

y = 1.0 / (1.0 + exp(-ax - b))

after observing the following data examples (x, y), the parameters a and b need to be updated using gradient descent with a learning rate = 1.

1. input: x = 1, y = 1, a = 1, b = 1

expected output: a = 0.881, b = 0.881

2. input: x = 2.2, y = 0, a = 5.1, b = 5.7

expected output: a = 7.3, b = 6.7

UPDATE

def update(x, y, a, b):
    t = 1 + np.exp(-a * x - b)
    da = (y - 1 / t) * x * (t - 1) * t**-2
    db = (y - 1 / t) * (t - 1) * t**-2
    new_a = a - da
    new_b = b - db
    return new_a, new_b

This got me

update(1, 1, 1, 1)
(0.9874844578263232, 0.9874844578263232)

Did I miss something here?

FINAL

def manual_update(x, y, a, b):
  p = 1 / (1 + np.exp(-a*x-b))
  pdpda = x*np.exp(-a*x-b) / (1+np.exp(-a*x-b))**2
  pdpdb = np.exp(-a*x-b) / (1+np.exp(-a*x-b))**2
  pdLda = -y*pdpda/p+(1-y)*pdpda/(1-p)
  pdLdb = -y*pdpdb/p+(1-y)*pdpdb/(1-p)
  return round(a + pdLda, 3), round(b + pdLdb, 3)

this worked perfectly.

$\endgroup$

2 Answers 2

1
$\begingroup$

I believe the answer given by Anoop A Nair is wrong. It takes me few time to recognise that I fall in same trap like Anoop A Nair.

The question said it is Logistic Regression. So the loss function should be cross entropy. Shouldn't be Squared Error loss.

$\endgroup$
3
  • $\begingroup$ so are you saying that the loss functio is L = y^ - 1 / (1 - exp(-ax-b))? $\endgroup$ Commented Jan 19, 2021 at 20:12
  • 1
    $\begingroup$ I don't be sure about this. But I saw Andrew Ng's course, it say it should be: Loss = −(ylog(p)+(1−y)log(1−p)) $\endgroup$
    – NoahShen
    Commented Jan 20, 2021 at 2:44
  • 1
    $\begingroup$ yes it worked! thanks very much. For anyone who is interested, there is the link to Andrew's video. youtube.com/watch?v=TTdcc21Ko9A $\endgroup$ Commented Jan 20, 2021 at 5:00
0
$\begingroup$

The update rules are ;

$a := a - \alpha.\frac{\partial L}{\partial a}$

$b := b - \alpha.\frac{\partial L}{\partial b}$

Here L which is the Loss [Squared Error in this case ] is given by;

$L = \frac{1}{2}.(\hat{y} - \frac{1.0}{(1.0 + exp^{-ax + b})})^{2}$

So ;

$\frac{\partial L}{\partial a} = \frac{1}{2}.2(\hat{y} - \frac{1.0}{(1.0 + exp^{-ax + b})}).\frac{-1.0}{(1.0 + exp^{-ax + b})^{2}}.(-x).e^{-ax+b} $

$\frac{\partial L}{\partial b} = \frac{1}{2}.2(\hat{y} - \frac{1.0}{(1.0 + exp^{-ax + b})}).\frac{-1.0}{(1.0 + exp^{-ax + b})^{2}}.e^{-ax+b} $

So now guess an intial value for a and b. You can plug in the an x and y pair for and update the weights simultaneously. The thing to be careful is that when updating for a and b in an iteration use the previous a and b values. That is if you have updated your a prior to b dont use the updated value of a to update b.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.