0
$\begingroup$

Recently, I implemented the LR algorithm in Python. The main part of the code is as following(I didn't use mini batch in my code. Instead, I use the whole batch to compute gradients every time):

class Logistic():
def __init__(self):
    self.w = None
    self.lr = 10.
    pass
def train(self, xs, ys):
    m, n = xs.shape
    ones = np.ones([m, 1])
    xs = np.hstack([xs, ones])
    ys = np.expand_dims(ys, -1)
    self.w = np.ones([n+1, 1], dtype=np.float64) * 1.0
    epochs = 100
    for epoch in xrange(epochs):
        y_ = self.sigmoid(-np.dot(xs, self.w))
        # loss = -1.0/m * np.sum(ys * np.log(y_) + (1 - ys) * np.log(1 - y_))
        tmp1 = np.sum(np.log(y_[np.where(ys==1)]))
        tmp2 = np.sum(np.log(1 - y_[np.where(ys==0)]))
        loss = - (tmp1 + tmp2) / m
        print("epoch: %d, loss: %f" % (epoch, loss))
        print("y_: %f, %f" % (np.min(y_), np.max(y_)))
        grad =  np.sum((y_ - ys) * xs, axis=0) / m
        self.w -= self.lr * np.expand_dims(grad, -1)
        print("grad: %f, %f" % (np.min(grad), np.max(grad)))
        print("w: %f, %f" % (np.min(self.w), np.max(self.w)))
        print ""

The dataset I used is MNIST. I marked all digits 0 as class 0, and all other digits as class 1. Then I get this binary classification problem. I test my algorithm with many different learning rate, from 1e-6 to 10, and it turns out all of them produces good results(about 98% accuracy on test set). As far as I know, if the learning rate is to big, LR will not converge. But here although I used very big learning rate, the algorithm still converge to about 98% accuracy. Is there an explanation for this?

$\endgroup$
1
  • $\begingroup$ Since you're already printing these things out: are your weights and loss function actually converging? $\endgroup$
    – Ben Reiniger
    Apr 3, 2019 at 11:41

1 Answer 1

1
$\begingroup$

What is the ratio between 0's and the other numbers? If the ratio is too low, let's say 2%, if you classify everything as not 0 you will have high accuracy.

EDIT AFTER COMMENTS

Imagine there are 98 non zeroes and 2 zeroes, you don't need any classifier to have a 98% of accuracy...

Accuracy is not the best metric for unbalanced data-sets, let's try precision and recall and take a look to the confusion matrix

$\endgroup$
9
  • $\begingroup$ About 10% 0's and 90% other digits. $\endgroup$ Apr 2, 2019 at 11:30
  • $\begingroup$ As I figured out, you have a very unbalanced data-set... Try to get rid off the 90% of the other digits, train again the algorithm, and let's see what happens... $\endgroup$
    – ignatius
    Apr 2, 2019 at 11:34
  • $\begingroup$ Is the ratio the same in the training test than in the test set? $\endgroup$
    – ignatius
    Apr 2, 2019 at 11:37
  • $\begingroup$ Yes, It's the same. $\endgroup$ Apr 2, 2019 at 11:38
  • $\begingroup$ I didn't take a look to your implementation, but the first steps should be balancing the data-set and using other metrics. $\endgroup$
    – ignatius
    Apr 2, 2019 at 11:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.