How to quantitatively evaluate raw neural network activations?

Question

Below are the activations for 2 different predictions. These predictions are for different labels/classes. They are being run through a dense Keras NN (96.6% accurate) with 2 hidden layers and adam optimizer.

What kinds of trends in the numerical values of the activations should I be looking for with regard to a NN that is being trained successfully?

• Do you want most activations to be either high or zero... not somewhere in the middle?
• What values would indicate that you need to add/ remove a layer?
• What maximum activation value is too high? Like if I saw an 8 should I be worried when the rest of my high values are around 4?

---

Answer 1

1) It is difficult to interpret a given weight on a given node for a given instance, especially in hidden layer.

2) You can only deal with specifical cases :

• Weights being very low for a given variable for all instance on the first layer means that you can remove the variable (If the variable is relatively uniform, standardized).

• For some activation functions, you can remove 'dead' neurons that activate to few (or no) instances. More generally, regarding to your questions, answers highly depends on the kind of activation. I would reccomend you to read : http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf by Lecun to avoid common traps. It will also give you some idea about what problems to look for (vanishing / exploding gradient, dead neurons... etc.)

3) On the architecture : as a rule of thumb for the size of your networks I would advise to start big, get a performance benchmark and remove layers / units untill the performance drops. I am not sure if you plotted only hidden layer or not, but in most cases the numbers of layers shoud decrease.

4) You might want to look some explanatory tools, like DEEPLIFT or SHAP, that will give you variable importance, summarizing all variable impact for an instance. These methods have some limitations but they will be easier to read.