I'm trying to understanding the relationship between training accuracy and training loss in classification tasks, specifically using logistic regression. When using logistic loss as the loss function and with the threshold set to $0.5$, I can see that the training accuracy and the total training loss both increase or decrease in the same direction. My reason for this conclusion is as follows. Denote by $m\in\mathbb{N}, \mathbf{w}, \mathbf{x}\in\mathbb{R}^n$ and $y\in\lbrace -1 , 1\rbrace$ the training sample size, the model's weight vector, some abstract (arbitrary) training example and its true label respectively. Let $\phi$ be the sigmoid function, $h_\mathbf{w}=\langle\mathbf{x}, \mathbf{w}\rangle$ be the net output function with weight vector $\mathbf{w}$. With threshold set to $.5$, the threshold function is as follows,
$$ f(\mathbf{x}) = \begin{cases} 1\text{ if }\phi(h_\mathbf{w}(\mathbf{x}))\ge 0.5 \\ 0\text{ otherwise}. \end{cases} $$
Also rewrite the binary cross-entropy loss function as follows,
$$ \mathscr{l}(h_\mathbf{w}, (\mathbf{x}, y))=log(1+e^{-y\langle \mathbf{w}, \mathbf{x}\rangle}), $$
and so the training loss would be
$$ \frac{1}{m}\sum_{i=1}^{m} log(1+e^{-y_i\langle \mathbf{w}, \mathbf{x_i}\rangle}), \text{ where }\mathbf{x}_i, y_i\text{ are a training example and its true label respectively}. $$
Let $P_w$ be the set of all predictions on the training sample $S$ made by the trained model $h_\mathbf{w}$ that are correct, that is
$$ P_\mathbf{w}=\lbrace(\mathbf{x}, y)\in S: y\langle \mathbf{w}, \mathbf{x}\rangle\ge 0\rbrace. $$
Because log is a monotonic function, so the loss decreases implies that the length of $|P_\mathbf{w}|$ increases. This along, with the threshold function defined above, in turn implies that the model's accuracy also increases.
First I'm not sure if there's any hole in my reasoning?
