# Understanding a Neural Network with Keras (preferably), TensorFlow or PyTorch

I would like to try a technique I saw in an article a while ago where, in order to understand what each neuron is doing, you apply specific inputs to the net and see which one of them is most activated.

Is there a simple way to do it with Keras (preferably)? I can take the weights of each neuron after the NN has been trained and use forward (predict) functions to activate it, but how would I get the activation result of each individual neuron? Is there a function for that?

TensorFlow and PyTorch offer more low level functions so I could do the same thing if I programmed the NN almost from scratch, but even so, does anyone know a simpler way?

If you have tips for this task also, they would be more than welcome. I don't know if functions for this kind of task already exist, but, if they don't, I would say it would be worthwhile creating them, people on the market have complained often to me that the main reason why they are not yet using NNs is that they don't know how to understand the NN's thinking.

• Could you give the link of this article in order to look at the algorithms and help you building it – LaSul Jan 9 '19 at 19:47
• I don't know if it will be that helpful, they used CNNs instead of NNs, so it is much more complicated there. What I had in mind was just simple NNs. Anyway, the article is Visualizing and Understanding Convolutional Networks, by Zeiler and Fergus. – Philippe Fanaro Jan 9 '19 at 20:09
• You can use tensorboard with keras, and get prettu much a deep insight about your network. Apart from that, here's a real fun tool to visualize a neural network in 3D: tensorspace.org/html/docs/startHello.html – Syed Ali Hamza Jan 9 '19 at 20:24

One simple way to do it, is to use the weights of your model to build a new model that's truncated at the layer you want to read. Then you can run the ._predict(X_batch) method to get the activations for a batch of inputs.