Would a neural network trained on extracted features have the same accuracy as a full network with frozen layers?

Question

Let's say that I train two neural networks on the exact same dataset.

1. The first network is a VGG19 model with frozen convolutional layers so only the top dense layers are getting optimized.
2. The second network is composed only of dense layers, with exactly the same architecture and hyper-parameters as the first networks fully connected layers.

Basically, I pass all my data through the convolutional layers of VGG19, saving outputs of the last convolutional layer to disk. Then, I load the data into my second model which is only composed of FC blocks and I train it to be a classifier.

Would combining the two models after training be the same as training a whole VGG19 model (with frozen conv layers) with my own FC layers?

If I understand correctly, the outputs of a trained network are entirely deterministic so in theory it should work. In practice, I have no idea if optimization algorithm takes frozen layers into account when training only the last FC layers.

Answer 1

When layers are "frozen" it generally means that their weights are not updated when backpropagation happens. So, technically, there is no difference between:

• Using a "frozen" VGG16 and training some fully connected layers and
• Using the VGG16 embeddings and training some fully connected layers

In practice, if you did both and compared the results, you may see differences if you haven't accounted for the fully connected layers' random initialisation.