AutoEncoder Architecture

My question is related with model-weights optimization during back propagation. In this image I'm trying to represent an auto-encoder having 7 layers where 4th one is center layer.

If my understanding is correct, during back propagation outcome of jth+1 layer, will help in optimizing the weights of jth layer. If it is correct, I want to restrict the weights training of Encoder layer and want to perform my customized training.

Customized training should be like this:

  1. Till layer 4 normal back propagation.
  2. Layer 3 weights will get trained(optimized) using the output of 4th layer as mentioned in PINK arrow.
  3. Layer 2 weights will get trained with the outcome of layer 5 as mentioned in YELLOW color arrow.
  4. Layer 1 weights will get trained with the outcome of layer 6 as mentioned in BLUE color.

I want to know how I can implement such customized trainable architecture?I'm familiar with Tensorflow, but I'm not able to figure out any solution and kind of stuck.

NOTE: Bear with my drawing.

  • $\begingroup$ I don't see a question here. We're a question-and-answer site, so we require you to articulate a specific, answerable question. "I need your support..." is not a question. $\endgroup$ – D.W. Feb 7 at 8:57
  • $\begingroup$ Just out of curiosity... WHY? $\endgroup$ – Leevo Feb 7 at 9:37
  • $\begingroup$ @Leevo, Bottleneck layer i.e. layer-4, which is very compact in size as compare to other layers and via this layer, other layers parameter are getting tuned. Its my assumption that, I can get a better results if I follow the approach that Im trying to implement. Hoping that able to answer your question. $\endgroup$ – vipin bansal Feb 7 at 10:18

You have to create multiple training sessions, in which you selectively freeze the layers you don't want to train. Each Sequential() model can be seen as a list of layer objects. Each of these layers have the argument trainable = True/False. You Freeze the layers you don't want to be trained classically, and proceed to

Then, using Keras Models, you connect layers that are not adjacent as you depicted in the picture. Once all layers have been trained, you transfer the weights into a news Sequential() shell to compose a final model.

It's going to be time consuming, but it's doable.

| improve this answer | |
  • $\begingroup$ More or less follow the similar approach using Tensorflow, but somehow it was not working. Let me re think about it. Anyways thanks for your comment. $\endgroup$ – vipin bansal Feb 7 at 10:24
  • $\begingroup$ NP, if you post some code chunks we might help you more $\endgroup$ – Leevo Feb 7 at 10:59

The definition of your customized training is too vague to be able to implement it: what is the "outcome" of a layer? How do you "train a layer with the outcome of another layer"?

Also, note that, while it is possible to freeze layers selectively during training (i.e. not applying any update during the optimization step), the gradients will always propagate backward through layers that are connected.

Nevertheless, there is a viable approach that may be somewhat similar to the intuition behind your proposal: residual connections.

enter image description here

As shown in the figure above (taken from the original paper), residual connections are simply connections from early layers to further points in the network by simply adding them. They allow gradients to be more easily propagated.

You could create your autoencoder so that the result of each decoder layer gets added the result of the encoder layer of matching size.

| improve this answer | |
  • $\begingroup$ As per my understanding, outcome of layer-7 should be close with layer-1. Similarly I'm assuming layer-2 should be close to layer-6 and so on. Even if my assumption is incorrect, I want to validate this scenario. Also during back propagation layer-3 will help in optimizing the weight parameters of layer-2 and layer-2 will help in optimizing the weights of layer-1. As I am assuming layer-1 is close to layer-7, layer-2 will be close with layer-6 and son on, that's the reason, I want to optimize weight parameter of layer 1 using layer 6 and so on. I believe that Im able to answer your question. $\endgroup$ – vipin bansal Feb 7 at 10:06
  • $\begingroup$ Please define "outcome of a layer" and "train layer X with the outcome of layer Y". These are the missing pieces of information to understand what you actually want to do. $\endgroup$ – ncasas Feb 7 at 10:08
  • $\begingroup$ Residual or skip level connection, Im not sure how it will help in implementing this solution. Because back propagation will help in similar classical way irrespective of introducing the outcome of previous layers to the some new layers or if Im missing something, please let me know. Still thanks a lot. $\endgroup$ – vipin bansal Feb 7 at 10:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.