I'm trying to understanding the relationship between training accuracy and training loss in classification tasks, specifically using logistic regression. When using logistic loss as the loss function and with the threshold set to $0.5$, I can see that the training accuracy and the total training loss both increase or decrease in the same direction. My reason for this conclusion is as follows. Denote by $m\in\mathbb{N}, \mathbf{w}, \mathbf{x}\in\mathbb{R}^n$ and $y\in\lbrace -1 , 1\rbrace$ the training sample size, the model's weight vector, some abstract (arbitrary) training example and its true label respectively. Let $\phi$ be the sigmoid function, $h_\mathbf{w}=\langle\mathbf{x}, \mathbf{w}\rangle$ be the net output function with weight vector $\mathbf{w}$. With threshold set to $.5$, the threshold function is as follows,

$$ f(\mathbf{x}) = \begin{cases} 1\text{ if }\phi(h_\mathbf{w}(\mathbf{x}))\ge 0.5 \\ 0\text{ otherwise}. \end{cases} $$

Also rewrite the binary cross-entropy loss function as follows,

$$ \mathscr{l}(h_\mathbf{w}, (\mathbf{x}, y))=log(1+e^{-y\langle \mathbf{w}, \mathbf{x}\rangle}), $$

and so the training loss would be

$$ \frac{1}{m}\sum_{i=1}^{m} log(1+e^{-y_i\langle \mathbf{w}, \mathbf{x_i}\rangle}), \text{ where }\mathbf{x}_i, y_i\text{ are a training example and its true label respectively}. $$

Let $P_w$ be the set of all predictions on the training sample $S$ made by the trained model $h_\mathbf{w}$ that are correct, that is

$$ P_\mathbf{w}=\lbrace(\mathbf{x}, y)\in S: y\langle \mathbf{w}, \mathbf{x}\rangle\ge 0\rbrace. $$

Because log is a monotonic function, so the loss decreases implies that the length of $|P_\mathbf{w}|$ increases. This along, with the threshold function defined above, in turn implies that the model's accuracy also increases.

First I'm not sure if there's any hole in my reasoning?

  • 1
    $\begingroup$ Please, clarify the following: 1) for binary cross-entropy, labels are in ${0, 1}$, why are you using ${-1, 1}$?, 2) the expression for the loss function does not look like it's correct (see this), where does it come from?. About your question about the categorical cross-entropy, you don't apply a threshold function but take the argmax. $\endgroup$
    – noe
    Nov 23, 2023 at 7:41
  • $\begingroup$ @noe, my bad it should be logistic instead of cross-entropy. Also I found the part I was wrong on the variation of the length of $|P_\mathbf{w}|$. However, after realizing that mistake, I still haven't figured out the relationship between the two. $\endgroup$
    – Tran Khanh
    Nov 23, 2023 at 9:23
  • $\begingroup$ It seems (though I haven't found solid examples) to me that the two quantities can vary in opposite directions and so you noted in the answer for my previous post. $\endgroup$
    – Tran Khanh
    Nov 23, 2023 at 9:25
  • $\begingroup$ I think there is a misunderstanding here. What do you mean by "it should be logistic instead of cross-entropy"? "Logistic" refers to the model not the loss; the loss of a logistic regression model is the binary cross-entropy, and the labels are in $\{0, 1\}$. $\endgroup$
    – noe
    Nov 23, 2023 at 13:31
  • $\begingroup$ @noe I mean the logistic loss function as defined in the question (which also is equal to cross-entropy for $\{0, 1\}$ labels) but since the labels are $\{-1, -1\}$ I need to switch back to logistic loss. Actually I didn't know that cross-entropy is restricted to 0,1 labels. $\endgroup$
    – Tran Khanh
    Nov 23, 2023 at 13:50


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.