I am implementing logistic regression with stochastic gradient descent, but it is not working as expected. I've tried many epochs and different learning rates $\alpha$ but the probability of belonging to the correct class oscillates around $0.5$. I have got two classes denoted as $0,1$ and two features. Is this scheme correct?:

Initialize weights $w = [w_0, w_1, w_2]$ to some random number (I choose 1) and loop:

  1. Choose random training sample $x_i = [1, x_1, x_2]$.
  2. Calculate the probability of it belonging to class $1,$ $$P(\hat{y} = 1) = \frac{1}{1+e^{-w \cdot x_i}}$$
  3. Update the weights according to $$w = w + \alpha x_i(y_i - P(\hat{y} = 1))$$

This scheme makes sense to me since the update to the weights would be big if the probability of belonging to class $1$ was low, if the correct class was $1$ but I can't verify it more than that.

  • $\begingroup$ Instead of showing the formula you intended to implement, please show the implementation itself (a minimal code example that people can run). $\endgroup$ – Mathias Müller Feb 16 at 11:55

Hard to say without more detail, but isn't your update wrong? you need to subtract rather than add the gradient. Unless alpha is negative, this is wrong.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.