"Super" Optimizer concept

Question

I was wondering why there isn't a feature built into common-use ML libraries, like Keras, that plugs many different combinations of layers and nodes to multiple models and trains them simultaneously to single out the best NN architecture for your problem?

For example, given training data, validation data, and a loss function, it compares a model consisting of two hidden Dense layers with 256 neurons each, and another model consisting of two hidden Dense layers, the first with 256 and the second with 64. It would then save the model with higher accuracy according to the input loss function.

Does something like this exist already? I know something like this exists with SKLearn's GridSeachCV, but I wasn't sure if that's a common practice outside of SKLearn. I feel like I might be oversimplifying a complex problem.

Thank you!

Answer 1

You could have a look at the AutoML work, which offers a few different ways to optimise a model of parameter space and goes well beyond Sklearn. They even have a tool that wraps around Sklearn!

Here is a summary from their homepage:

Here is a longer description, and here is a full example based on Keras.

These libraries can essentially combine the best known practical methods of optimisation, such as:

• standard grid-search over parameters
• Bayesian optimisation of parameters
• a combination of the two methods: HpBandSter (HyperBand on STERoids)

Answer 2

There's also open-source NAS AutoKeras. To answer your question: there's none such functionality in popular libs, as NAS is still in its infancy. Searching this space using metaheuristics is still in research phase and please note - this space is huge. Most of the people cannot afford it hardware-wise. It'll take some time till we got reasonable evo NAS and evo net optimizers. It might also turn out it doesn't make much sense.