# Auto-Encoder customized layer training

My question is related with model-weights optimization during back propagation. In this image I'm trying to represent an auto-encoder having 7 layers where 4th one is center layer.

If my understanding is correct, during back propagation outcome of jth+1 layer, will help in optimizing the weights of jth layer. If it is correct, I want to restrict the weights training of Encoder layer and want to perform my customized training.

Customized training should be like this:

1. Till layer 4 normal back propagation.
2. Layer 3 weights will get trained(optimized) using the output of 4th layer as mentioned in PINK arrow.
3. Layer 2 weights will get trained with the outcome of layer 5 as mentioned in YELLOW color arrow.
4. Layer 1 weights will get trained with the outcome of layer 6 as mentioned in BLUE color.

I want to know how I can implement such customized trainable architecture?I'm familiar with Tensorflow, but I'm not able to figure out any solution and kind of stuck.

NOTE: Bear with my drawing.

• I don't see a question here. We're a question-and-answer site, so we require you to articulate a specific, answerable question. "I need your support..." is not a question.
– D.W.
Feb 7 '20 at 8:57
• Just out of curiosity... WHY? Feb 7 '20 at 9:37
• @Leevo, Bottleneck layer i.e. layer-4, which is very compact in size as compare to other layers and via this layer, other layers parameter are getting tuned. Its my assumption that, I can get a better results if I follow the approach that Im trying to implement. Hoping that able to answer your question. Feb 7 '20 at 10:18

You have to create multiple training sessions, in which you selectively freeze the layers you don't want to train. Each Sequential() model can be seen as a list of layer objects. Each of these layers have the argument trainable = True/False. You Freeze the layers you don't want to be trained classically, and proceed to

Then, using Keras Models, you connect layers that are not adjacent as you depicted in the picture. Once all layers have been trained, you transfer the weights into a news Sequential() shell to compose a final model.

It's going to be time consuming, but it's doable.

• More or less follow the similar approach using Tensorflow, but somehow it was not working. Let me re think about it. Anyways thanks for your comment. Feb 7 '20 at 10:24
• NP, if you post some code chunks we might help you more Feb 7 '20 at 10:59

The definition of your customized training is too vague to be able to implement it: what is the "outcome" of a layer? How do you "train a layer with the outcome of another layer"?

Also, note that, while it is possible to freeze layers selectively during training (i.e. not applying any update during the optimization step), the gradients will always propagate backward through layers that are connected.

Nevertheless, there is a viable approach that may be somewhat similar to the intuition behind your proposal: residual connections.

As shown in the figure above (taken from the original paper), residual connections are simply connections from early layers to further points in the network by simply adding them. They allow gradients to be more easily propagated.

You could create your autoencoder so that the result of each decoder layer gets added the result of the encoder layer of matching size.

• As per my understanding, outcome of layer-7 should be close with layer-1. Similarly I'm assuming layer-2 should be close to layer-6 and so on. Even if my assumption is incorrect, I want to validate this scenario. Also during back propagation layer-3 will help in optimizing the weight parameters of layer-2 and layer-2 will help in optimizing the weights of layer-1. As I am assuming layer-1 is close to layer-7, layer-2 will be close with layer-6 and son on, that's the reason, I want to optimize weight parameter of layer 1 using layer 6 and so on. I believe that Im able to answer your question. Feb 7 '20 at 10:06
• Please define "outcome of a layer" and "train layer X with the outcome of layer Y". These are the missing pieces of information to understand what you actually want to do.
– noe
Feb 7 '20 at 10:08
• Residual or skip level connection, Im not sure how it will help in implementing this solution. Because back propagation will help in similar classical way irrespective of introducing the outcome of previous layers to the some new layers or if Im missing something, please let me know. Still thanks a lot. Feb 7 '20 at 10:27