I found online that the derivative of a cross-entropy activation function with a softmax activation is (output - expected), which had me very confused. If for example, the expected value is 1, and since the output is always less than 1 (softmax), doesn't it mean that the derivative is negative, and that if we train our weights with this gradient, the weights will decrease, and so the output will decrease in the next iteration, thus increasing the error? How does it work?
As you mentioned, (output-expected) see, this value will be less than 0 (negative as output<1 and expected=1) Now w_new = w_old - learning_rate*derivative now based on derivative, w_new can be greater or lesser than w_old.
I guess when you are assuming with negative derivative, weights decrease is not correct. Also even if weights decrease the output will decrease is also not correct.