I got tasked with reproducing the results of the influential "How transferable are features in deep neural networks?" paper in a DL class I'm taking (Full code).

I got the exact opposite results (left are original paper's results, right are mine), which obviously doesn't make sense:

enter image description here

As can be seen above:

  • Transferring from A to learn B: Original paper shows degrading results the more layers are being transferred, while I observe increasing performance.
  • The above is also true for transferring from B to A, but the original paper doesn't include this result.
  • Transferring from B to B: Original paper shows a convex pattern, while for me it's mostly concave.

I'd like to understand what could be the fundamental reason that might have caused that.
Here is the core configuration of my experiments:

  • A small subset of CIFAR10, with about 500 train and 100 test images for datasets A and B each, randomly sampled and selected by label.
  • CIFAR10 classes are split into two, such that model A and model B have 5 classes each.
  • VGG16 architecture for model A and B.
  • Layer splits were done in the MaxPooling layers.

Assuming there's no bug in my code (which is an option, but doesn't seems like it (again, full code), I guess the only options are either dataset size or network depth, but I can't really hypothesize in favor of one them.

How to make sense of these results?

  • $\begingroup$ The x-axes are completely different, why is that? $\endgroup$ Apr 11 at 11:39
  • 1
    $\begingroup$ @picky_porpoise, my own x-axis (right side) lists the actual VGG16 layer's index,, having 30 layers in total. The original paper's x-axis (left side) lists an "aggregated" layer index, i.e. it refers to a combination of conv + relu + pool layers as a single layer. This is not stated explicitly in the paper, but it can be viewed in the original paper Github (here's one example): The results folder lists the many networks they used, each having files named prototxt files with the architecture. $\endgroup$
    – OfirD
    Apr 12 at 0:07


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.