Combining CNNs for image classification

I would like to take the output of an intermediate layer of a CNN (layer G) and feed it to an intermediate layer of a wider CNN (layer H) to complete the inference.

Challenge: The two layers G, H have different dimensions and thus it can't be done directly. Solution: Use a third CNN (call it r) which will take as input the output of layer G and output a valid input for layer H. Then both the weights of layer G and r will be tuned using the loss function:

$$L(W_G, W_r) = MSE(\text{output of layer H}, \text{output of r})$$

My question: Will this method only change the layer G's weights along with r's weights? Does the whole system require finetuning afterwards to update the weights of the other layers?

It is straightforward to stack different neural networks with current deep learning frameworks (i.e., PyTorch or TensorFlow). There is no separate output for $$r$$, it just one of many layers.

It could look like this:

You can freeze or not freeze any layers in a stacked neural network. You can decide how far to backpropagate the training updates.

The advantages of unfreezing layers is that the model can learn better feature representations. The disadvantages of unfreezing is that training can take longer for very deep neural networks.

• Can you please elaborate on how this stacked neural network architecture should be for my problem? How $r$ ends up being one of many layers? I've never encountered stacked NN's before and I have a hard time grasping it. Jul 31 at 18:18
• Sure - I added a picture to illustrate the architecture. Jul 31 at 20:48
• Thank you. So, one final question to fully understand the practical aspect of it. We take the first (narrow) CNN up to the intermediate hidden layer $G$, then stack sequentially the whole $r$ and finally stack the second (wider) CNN from the intermediate hidden layer $H$ up to its softmax output. Is that description accurate? Aug 1 at 9:05

First of all: probably you should not train with the loss you propose, because with MSE you will train to minimize the total error, not to keep the features as they were, which is what CNNs are good at detecting; this is the same problem as what happens when you train an image autoencoder on MSE, that you obtain blurry images. Instead, configure the network as you want, reusing layers from the networks you deem appropriate, with the needed adaptation layers, and then train the whole network on the task that you need your network to do (e.g. classification).

When doing so, you can choose to only train some parts of the network, or train them at different learning rates. These are some potential alternatives:

1. Freeze all the layers except the adaptation ones. The original weights of the reused layers will remain as they originally were.
2. Train the adaptation layers at a normal learning rate and the other layers at a very small learning rate. This is typical of transfer learning setups and it aims at making the learning more flexible.
• Thanks for your answer. Is it possible for you to briefly define what you mean by "reusing layers" and "adaptation layers"? I have a hard time understanding what architecture would be suitable for my problem. Jul 31 at 18:26
• With "adaptation layers" I mean what you called "r", that is, the layers you add to bridge the output of layer G and H. With "reusing layers" I mean using the weights of the layers of G and H into a new neural network, together with r to bridge them
– noe
Jul 31 at 18:52