I'm interested in doing research similar to the work done in this blog, The Unreasonable Effectiveness of Recurrent Neural Networks by Andrej Karpathy. Specifically, where the author explores the rules that individual neurons have learned by extracting the activations or excitement of random neurons at the prediction step and visualizing them as a heatmap on top of the predicted text.
I'm having a hard time figuring out how to do this with existing platforms such as TensorFlow. My first strategy was to try a form of transfer learning and cut off a copy of the network at an activation layer and have that layer's activations output side by side with a text prediction. But the weakness of this approach is that I can't actually see the activations in the hidden state (or more than one layer at a time), just the outputs of an LSTM layer.
Can anyone recommend a strategy or starting point where I can reveal the activations of neurons in the hidden states of RNNs (LSTM or GRU), or if there's a way to do this in TensorFlow that I've overlooked? If I build the network from scratch with TensorFlow, is there a way to efficiently set it up to output this data on prediction without significantly adding to the computation expense during training? I couldn't find the answer in the source code.
Can anyone give me a nudge in the right direction?
Thanks in advance!