1
$\begingroup$

I am doing deep learning binary classification on some data and got very weird results with the accuracy metric. In the first few epochs, it doesn't change at all but then it goes on this weird linear path. I have attached a picture below. Can someone tell me what this means since I am new to machine learning and I am used to nice logarithmic graphs? enter image description here

$\endgroup$

1 Answer 1

1
$\begingroup$

Assuming that the dataset is balanced, my intuition is the following:

From epoch 1 to 55: the loss function being superhigh indicates your model is doing random predictions but with probabilities near 0 or 1. This is, to each example it assigns randomly a value near 0 or 1. The log-loss formula is

$$ \mathcal{L(y_i, p_i)} = - \left[y_i \log p_i + (1-y_i) \log(1-p_i) \right] $$

If the real label is $y_i=0$ and your wrong prediction is $p_i \to 1$, then the loss function is $\mathcal{L}(y_i, p_i) \to \infty$. Also, the random predictions explain the 50/50 accuracy (assuming a balanced dataset). During these epochs, your model is not learning, but just calibrating the predicted probabilities to be in a more reasonable range.

From epoch 55 to 300: After epoch 55 seems that your model starts to learn. This is also reflected in your accuracy plot, where the accuracy starts to improve. In your loss plot, it seems the loss is not changing, but this may be an illusion. Try to change the y-range to (0, 1) and I guess you'll see your loss decreasing.

My recommendation is to be careful in the way you initialize your network since they have a lot of impact on your network learning process. There are a lot of resources about this topic, like this one.

$\endgroup$
1
  • $\begingroup$ Thank you it makes sense now. I also tried it with 1000 epochs, and eventually, the accuracy becomes a nice logarithmic curve. And yes the loss decreases even more but is not shown in this graph. $\endgroup$
    – Dani
    Jun 16, 2022 at 9:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.