Choosing an optimizer to perfectly fit a neural networks to training data

Question

In short, my query is: Which optimizer(s) should one choose to experiment for a fully connected neural network if she wants perfect fitting (mae < 1e-04) on the training data?

Details: In my particular case, the input of the function is 60-dimensional and the output is 1-dimensional (the training data set is prepared by solving a forward model). The input and output are normalized and the sample mean is converted to 0. My neuron's activation function is tanh (this gave me a better result than ReLu and Sigmoid). I have so far used Keras's "adam" and "sgd" as optimizers and I have tried with various learning rates.

In addition, I have tried with increasing the number of neurons in each hidden layer and increasing the number of hidden layers as well. At some point, the total number of my trainable parameters was more than 100 million.

However, even for trying different batch sizes and 10,000 epochs, my best mean absolute error (mae) was never below 0.07. The only part I haven't yet experimented much is to customize/use advance optimizers. I am completely clueless why the neural network can't find a set of trainable parameters where, at least, it can over-fit the training data?

Any suggestions from the experts? Thanks in advance for your time and patience. I really appreciate your support.

An example code:

input_size = ND
nodes = 10000
inp = Input(shape=(input_size,),name='Input')
l0 = Dense(nodes, activation='tanh',name='Level0')(inp)
l1 = Dense(nodes, activation='tanh',name='Level1')(l0)

lo = Dense(1, activation='tanh',name='Level_out')(l1)

merged = Model(inputs=inp,outputs=l10)
opt = optimizers.adam(lr = 1e-5)
merged.compile(optimizer=opt,
              loss='mean_squared_error',
              metrics=['mae'])

Answer 1

There's no free lunch, only bunch of tips. Your search space always depends on the problem you're tackling, and there's no one that fits them all. You need to run experiments to check this. Additionally, your net's architecture is one of the parameters, so it's hard to tell if your net will even be of sufficient capacity, without knowing the exact functions, you want to approximate. At first glance, optimizer is last thing you should worry about.

Answer 2

I agree with @Piotr Rarus (+1); as he said, the no free lunch theorem certainly applies to this.

I'd like to add some exploration of the four possible states for perfectly fitting the training data:

1. Impossible due to training function if it is only surjective (onto but not one-to-one). This is of course unlikely, but still feasible.
2. Impossible due to network architecture (e.g. with a single layer, you could not fit nonlinear functions)
3. Possible and trivial (e.g. convex problem space).
4. Possible and hard but many local minima and/or saddle points. Possible but difficult to prove, or even attain within some tolerance, global optimality.

Given that you had difficulty reaching low error with local optimizers, it's likely that you are in state 4.

Deterministic global optimization is generally quite difficult, but you have some tricks at your disposal. Firstly, if you attain an error of 0, then you know you've found a non-strict global minima.

With sufficient computational power, you could try some of these deterministic global optimization methods, or you could even brute-force it by splitting the search-space into a hypercubic lattice, defined by your tolerance, and grid search the parameters.

However, assuming you do not have such massive computational resources, and assuming you want to keep this network architecture, you could use more classical techniques for overcoming local minima, such as momentum. Some other classical methods are higher-order, such as the Quasi-Newton method. Unfortunately, given the status quo of deep learning, stochasticism and local optimization is still very much ingrained in network training, particularly in popular frameworks/libraries.

Lastly, I'd recommend adding more layers. I don't know how you distributed the 100+ million parameters, but depending on the training set function, it may not have been sufficiently nonlinear.

Answer 3

It sounds like you’re not trying to create a model that can be used outside of the training dataset and instead you’re trying to get your network to memorise the dataset. In other words, you’re aiming to overfit.

If that is the case the simplest thing that might work is to take the group of samples your network is getting wrong and upsample them (include them more times in your training data) until the error they add to your training is sufficient to get you out of the local minimum you’re currently getting stuck in.