I try a simple Neural Network (Logistic Regression) to play with Keras. In input I have 5,000 features (output of a simple tf-idf vectorizer) and in the output layer I just use a random uniform initialization and an $\alpha = 0.0001$ for $L_{2}$ regularization with a sigmoid activation function (so something pretty standard).
I use an Adam optimizer and the loss function is the binary cross-entropy.
When I display the probability for both classes I end up with something like this :
After that I try to add a hidden layer with a $Relu$ activation function, 64 nodes and again the same parameters for regularization & initialization. -- EDIT -- Here for the output layer I keep exactly the same parameter (and $sigmoid$ activation function) as in the previous NN. -- END OF EDIT --
But now when I plot the probability distribution for both classes I end up like this :
And cleary I didn't get why by adding a hidden layer the probability are push to 0 or 1 ?
Have you any references that I can read so I can understand the math behind ? It could be great!
Moreover sometimes (with a more "deeper" neural network for another application) I get the same plot as the second one but this time the predicted probabilities are between $[0.2; 0.8]$. The probabilities are push to some "values" but those values are more "centered". Not sure it is clear. And here again I don't understand what is the phenomena behind this ? How can I "debug" it to see in my architecture of my Neural Network what is the cause ?
In addition How can I "tweak" my ANN to have a "perfect" calibration plot (like in the scikit webpage : https://scikit-learn.org/stable/modules/calibration.html) ?
Thank you in advance for every answer that can enligthen me :)