Having weird accuracy graph on deep learning binary classification model

Question

I am doing deep learning binary classification on some data and got very weird results with the accuracy metric. In the first few epochs, it doesn't change at all but then it goes on this weird linear path. I have attached a picture below. Can someone tell me what this means since I am new to machine learning and I am used to nice logarithmic graphs?

Answer 1

Assuming that the dataset is balanced, my intuition is the following:

From epoch 1 to 55: the loss function being superhigh indicates your model is doing random predictions but with probabilities near 0 or 1. This is, to each example it assigns randomly a value near 0 or 1. The log-loss formula is

$$ \mathcal{L(y_i, p_i)} = - \left[y_i \log p_i + (1-y_i) \log(1-p_i) \right] $$

If the real label is $y_i=0$ and your wrong prediction is $p_i \to 1$, then the loss function is $\mathcal{L}(y_i, p_i) \to \infty$. Also, the random predictions explain the 50/50 accuracy (assuming a balanced dataset). During these epochs, your model is not learning, but just calibrating the predicted probabilities to be in a more reasonable range.

From epoch 55 to 300: After epoch 55 seems that your model starts to learn. This is also reflected in your accuracy plot, where the accuracy starts to improve. In your loss plot, it seems the loss is not changing, but this may be an illusion. Try to change the y-range to (0, 1) and I guess you'll see your loss decreasing.

My recommendation is to be careful in the way you initialize your network since they have a lot of impact on your network learning process. There are a lot of resources about this topic, like this one.