Say I have fine-tuned a 10-classification ResNet18 network on CIFAR-10 and the accuracy on validation set is about 93%.

However when feeding into 5000 random noise images (Gaussian noise with the same mean and stddev of CIFAR-10), not as expected to be randomly sampled into 10 class, nearly all of them are classified as bird:

noise's prediction distribution

Then I examine the feature vector after the last conv layer (before FC layer), with one image corresponding to a 512-dim vector. For each of the 11 categories (CIFAR10+noise), I use the mean of all the 512-dim vectors of images under this category to represent this category. Cosine between two categories' representing vectors is used to measure the DISTANCE between two categories. ($d(v_1,v_2)=\frac{1}{2}(1-cos(v_1,v_2))$ so that the less $d(v_1,v_2)$ is, the closer $v_1$ and $v_1$ are) Then create a graph with 11 nodes, each two nodes are connected with an edge weighted $d(v_1,v_2)$. The MST of this graph: enter image description here Distance between bird and noise is exactly rather small.

This also happens if the network is only 2-classification trianed on cats and dogs. For the rest planes, planes, frogs, etc., over 70% of them are misclassified as cat, except for horses that 72% of them are classified as dog. As for random noises, 100% of them are classified as cat.

So some natural questions come to me :

  1. If it's that cat images in training set are more clutter than that of dog? And bird is the most clutter among the 10 categories?
  2. How can 'clutter' be mathematically represented? Spanning in a larger dimension space? Using manifold, PCA or K-means to calculate distance from centroid to bound? Over original images or feature maps?...
  3. Maybe that birds in training set images always appear with a noisy background? So noise is exactly contributing positively for bird?
  4. How to pull out what a neuron in the network really learned to classify the 10 categories?

I searched something about this and maybe it's called Out-of-Distribution Detection? What I'd like to do now is not to boost the accuracy but to figure out why this happens in the interpretable view.

Could you please give me some suggestions on my thoughts listed above or the further experiments designing? Some papers related to this is also appreciated.

Big thanks!

  • $\begingroup$ Some suggests that "Random noise isn’t going to have a strong presence of any features so it’ll mostly come down to the bias terms" (for $x \in Noise, y(x)=wx+b≈b$). However in the last layers 10-dim bias, BIRD doesn't have the highest bias. $\endgroup$
    – dmrak
    Dec 28, 2019 at 10:32
  • $\begingroup$ Also asked here. To ask it again in Data Science chart, I'd like to know more from the view of using clustering, manifold, etc., to interpret the problem. thx. $\endgroup$
    – dmrak
    Dec 28, 2019 at 10:34


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.