I understand how to build and train a neural network like shown below, as well as those low-level features/filters. I wonder what are those high-level features: how exactly do you obtain them from a trained neural network? (Are those like the "eigenfaces"?)


Note: the image is by NVIDIA, and I don't know the specifics of the classification problem here. If needed, suppose the network is trained to distinguish human from cat.


2 Answers 2


The image, and variants of it that are commonly used are for illustrative purposes only. They generally do not represent data that has been extracted from real CNNs.

The first "Low-level features" part of the diagram is possibly from a real network (I am not sure in this case, it looks more like a constructed filter, e.g. Sobel, to me). That is because it is feasible and relatively easy to interpret the first layer's filter weights directly as images, and the filters do indeed look like the components that they detect.

The "Mid-level features" and "High-level features" in your specific diagram have probably been constructed without using a neural network. They are likely to be an artists impression of what the high level features might be. They may have been sampled from real datasets, then just cropped and arranged into the image.

Caveat: I cannot find absolute evidence for the specific image being constructed for illustration only, just I suspect this to be the case.

It is possible to extract visualisations of features detected by deeper layers. The two common ways to do this are:

  • Dataset matching. Finding examples in the dataset which trigger a specific neuron to output a high value. This can be isolated to a crop of the original image, because you know the combined sizes of all the filters and pools that occur before layer you are interested in.

  • Optimising the input image. Using gradient ascent, but instead of changing the weights, make a cost function that scores the neuron you want to visualise and keep adjusting the input until

You can get more information from resources such as this article on feature visualisation.

  • $\begingroup$ This is sufficiently exact for me -- thanks, especially for the reference. If you know of other, less common ways, please feel free to expand your answer. $\endgroup$
    – user66081
    Jan 19, 2018 at 10:33
  • $\begingroup$ Is there a consensus on what is meant by "low-level" and "high-level" features? $\endgroup$
    – ado sar
    Jan 31 at 15:03
  • $\begingroup$ @adosar in terms of low-level meaning small groupd of pixels, and high-level meaning larger sections of recognisable objects, then yes this is the most commonly shared understanding. In terms of where boundaries lie, then no because these are slightly vague intuitive descriptions, not mathematically precise categories $\endgroup$ Jan 31 at 15:29
  • $\begingroup$ Can we also think of "high-level" features as feature capturing the "global" details (e.g. containg a face or not) of the image and not the "local" details such as the exact pixel values etc? Moreover, "high-level" are also called "abstract" features. What is meant by abstract? E.g. in the deeplearning book it is stated that more abstract concepts are defined in terms of simpler ones, but I fail to grasp it. I get that the first conv layer for exampe detect edges, the second might detect corners by combining the edges from first layer. $\endgroup$
    – ado sar
    Jan 31 at 20:08
  • $\begingroup$ I don't however understand what more "abstract" features or concepts means. Is it related to humans? $\endgroup$
    – ado sar
    Jan 31 at 20:13

Its about how the neural net learn inside. Usually in deep neural network you have multiple layers, the first layers will learn the low level feature then the more you approach the output layer the more the layers will learn the high level feature. Here its identifying faces task, that's why you see that in the first layers small patterns are learned (mainly edges ...) then faces components ...

How to obtain them : you should watch what makes neuron activated in each layer depending on the input. As you know each neuron will be activated (once the DNN is trained) for specific input combinations. By visualizing that you can get an idea about what exactly each layer has learned in term of high-low level features.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.