Let's say that I train two neural networks on the exact same dataset.

  1. The first network is a VGG19 model with frozen convolutional layers so only the top dense layers are getting optimized.
  2. The second network is composed only of dense layers, with exactly the same architecture and hyper-parameters as the first networks fully connected layers.

Basically, I pass all my data through the convolutional layers of VGG19, saving outputs of the last convolutional layer to disk. Then, I load the data into my second model which is only composed of FC blocks and I train it to be a classifier.

Would combining the two models after training be the same as training a whole VGG19 model (with frozen conv layers) with my own FC layers?

If I understand correctly, the outputs of a trained network are entirely deterministic so in theory it should work. In practice, I have no idea if optimization algorithm takes frozen layers into account when training only the last FC layers.

  • 1
    $\begingroup$ in theory it might work, although there are many parameters to take account of (eg finite precision effects, algorithms used, ..). But I miss the point of the question. If one of these approaches suits your goal, why baffled over such theoretical issues, that actual practice may render useless musings? $\endgroup$ – Nikos M. Mar 26 at 17:20
  • $\begingroup$ I want to make sure that the results would be identical no matter which approach I choose. Otherwise I couldn't say that I used VGG19 architecture in my paper. $\endgroup$ – FloppyC0de Mar 26 at 17:30

When layers are "frozen" it generally means that their weights are not updated when backpropagation happens. So, technically, there is no difference between:

  • Using a "frozen" VGG16 and training some fully connected layers and
  • Using the VGG16 embeddings and training some fully connected layers

In practice, if you did both and compared the results, you may see differences if you haven't accounted for the fully connected layers' random initialisation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.