In short, my query is: Which optimizer(s) should one choose to experiment for a fully connected neural network if she wants perfect fitting (mae < 1e-04) on the training data?
Details: In my particular case, the input of the function is 60-dimensional and the output is 1-dimensional (the training data set is prepared by solving a forward model). The input and output are normalized and the sample mean is converted to 0. My neuron's activation function is tanh (this gave me a better result than ReLu and Sigmoid). I have so far used Keras's "adam" and "sgd" as optimizers and I have tried with various learning rates.
In addition, I have tried with increasing the number of neurons in each hidden layer and increasing the number of hidden layers as well. At some point, the total number of my trainable parameters was more than 100 million.
However, even for trying different batch sizes and 10,000 epochs, my best mean absolute error (mae) was never below 0.07. The only part I haven't yet experimented much is to customize/use advance optimizers. I am completely clueless why the neural network can't find a set of trainable parameters where, at least, it can over-fit the training data?
Any suggestions from the experts? Thanks in advance for your time and patience. I really appreciate your support.
An example code:
input_size = ND nodes = 10000 inp = Input(shape=(input_size,),name='Input') l0 = Dense(nodes, activation='tanh',name='Level0')(inp) l1 = Dense(nodes, activation='tanh',name='Level1')(l0) lo = Dense(1, activation='tanh',name='Level_out')(l1) merged = Model(inputs=inp,outputs=l10) opt = optimizers.adam(lr = 1e-5) merged.compile(optimizer=opt, loss='mean_squared_error', metrics=['mae'])