I am testing different RL methods, and I know e.g that policy gradient method is supposed to have a high variance gradient which can cause trouble. I want to run a few different Deep RL algorithms, and see if I can see any patterns in the gradient. How can I visualize the gradient, and analyze it?
So far I have been using the average standard deviation, of the gradient, for every layer in my neural network and visualized it as a scalar with epoch on the x-axis. Is there a better way?