Highly accurate convnets appear to have random-looking visualized weights?

Question

I'm building a TensorFlow convoluted neural network that isn't getting the accuracy that I hoped for. So I figured I would visualize the learned weights to see where the network might be stumbling. As a benchmark I started visualizing the weights for a totally different project, Google's MNIST convnet example, which has very high accuracy (99.2%).

I assumed that a very accurate model would have intuitive weights, but in reality I got weights that looked completely random. Other people seem to get similarly random results. See martin-gorner's comment with visuals on Mar 18. His are results are similar to mine.

More Googling shows that other people see essentially random behavior for non-MNIST datasets. Is this common? If so, it would seem that visualizing weights is a fruitless exercise unlikely to bring any helpful insights to the modeler. Is this true? If not, why do so many people use this visual (clear edges in the first layer, and composite images in subsequent layers) when describing how convnets work? Stanford's course material also seems to at least show edges (you have to scroll down about half of the page) when visualizing the weights. I'm not sure what to think.

Answer 1

The weights learned have no straightforward interpretability and i would not interpret those to be "random". What exactly do you mean by "random"? They seem to be learning very basic patterns that might reasonably extract and combine into more complex patterns as you deeper in the network, which is exactly how the networks are supposed to work.

from the Keras blog post you linked,

A remarkable observation: a lot of these filters are identical, but rotated by some non-random factor (typically 90 degrees). This means that we could potentially compress the number of filters used in a convnet by a large factor by finding a way to make the convolution filters rotation-invariant. I can see a few ways this could be achieved --it's an interesting research direction.

The stanford page shows a great bit of what the network seems to be learning at an early layer -- combinations of edges and color blobs.

Perhaps this is more philosophical in nature, but why would we expect the layers to be meaningful or easy to interpret for humans? The network's huge number of weights are learned by optimizing a loss function via gradient descent. It can be difficult to interpret how far less complicated non-linear models "learn" or weight their coefficients; I'd expect these models would be (and are!) wildly difficult to interpret. The fact that we can see any structure in the various visualizations is cool, but they probably won't help us understand why a model isn't working all that well.

Regarding your problem, how many classes do you have and what type of accuracy are you hoping for? MNIST is an almost trivial dataset since the classes are very well separated. Some of the real-life machine vision tasks I deal with at work with good somewhat clean classes can get 80-90%~ accuracy, but errors are made in photos that humans might call something else (e.g., a photo of a studio apartment's living room in which the kitchen is clearly visible might be labeled as a kitchen and not a living room, but this is mostly okay). Defining what is a "good enough" accuracy is an issue on its own -- that's more than enough for what we're doing at the moment.

If I missed the point of your question, please let me know.