Convolutional Neural networks are used in supervised learning meaning models are always "set in stone" after training (architecture and paramters) so this might not even be possible, but is there any research done on playing around with the data paths, model size (number of layers) and architecture during runtime, i.e after training is done, for instance creating a model that can be modified online to use less or more recources, skip layers or use portions of the model.

There is some work done recently on creating flexible frameworks for training and designing networks, but that's always "offline". There is also an interesting paper on training two models, "big" and "little" for the same application and using an accuracy/power trade-off policy to deploy one of the two.

  • $\begingroup$ Resnet which has skip connection is somehow designed to do what you are referring. $\endgroup$ – Media Jul 10 '18 at 15:27

It's a great question and one I have thought about a lot too.

Are you are talking about a network structure that evolves/mutates during training, or one that simply chooses whether or not to use all available resources? While I would never give a straight-up No to such a questions (especially the former), I can't say I have really seen it in publications. The cloest thing that comes to mind is a recent paper from DeepMind, which utilises a combination of "model-free and model-based aspects"; however, this more high-level than network architectures specifically, as you refer to them.

Perhaps you are considering a model, which is forced to choose between one resource and another? An example of the latter would be whether or not to use a skip connection or a dense layer (assuming shapes all match). Yann LeCun says (the video linked below) that such paths in a network not only provide more flexibility/complexity, but additionally ease the optimisation problem and provide a source of regularisation.

In any case, I am sure people are doing that, making use of dynamic networks created using PyTorch, the control flow operators in Tensorflow, the new tensorflow eager library (which aims to replicate dynamic networks like PyTorch produces).

One could also argue that providing a monolithic network to a model affords it the flexibility to use the resources (i.e. the weights) that it chooses. It doesn't have to use them all. Just as batch normslisation layers allow the model to parametrically adjust the extent to which they are used.

There are great discussions on what is required in the long run, heading toward Artificial General Intelligence (a recent overview in the context of RL part1 & part2. If we should be training models end-to-end, how much information should we give an algorithm learning to play a game and

The tools that would allow people to do that, like you mention, are coming along, but I haven't yet seen a paper that explicitly outlines how that is done. It would be great is someone else gives you a more satisfying answer! ;-)

  • $\begingroup$ Thanks for all the links, I'm checking them now, the discussion between Manning and LeCun is especially interesting. My question was basically a combination of the two points you brought up, I have been looking for publications on this topic for a few days now and it seems like most of the work done in this area still results in "rigid" structures. $\endgroup$ – Alla Jul 10 '18 at 14:50
  • $\begingroup$ @Alla - I think part of the programme is a programming one... how can you make a model that mutates during training (or inference!) ? I think the closest research is maybe that looking at meta-models, which are perhaps more suited. They do, however, still have a fixed nature as far as I can tell. Check out this paper on Meta Learning with Shared Hierarchies as a starting point. $\endgroup$ – n1k31t4 Jul 13 '18 at 11:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.