Tuning the model parameters vs the parameter of optimizer for Deep Neural Networks?

Question

I understand that there are rarely general recipes in field of machine learning and the many results can be achieved only by trial and error, and are task specific as well.

My question is, if the model doesn't give a desired quality, but one may expect, that the model of this or similar architecture can achieve a reasonable result, based on some prior assumptions or knowledge, what is, in general, better strategy, more successful in majority cases:

• Tuning of the optimizer parameters of the optimizer. (playing with the learning rates, random seeds, weight decay, momentum, e.t.c). hoping that it converges to the better optimum
• Changing some architecture properties of the model (number of hidden layers, dropout rate, width, number of filters in CNN)

Here I assume, that the current network is sensible choice - not 1-hidden layer narrow MLP - for solving some complicated task.

Minimization of loss function doesn't always lead to improvement of the quality of the model on the test data (well known problem of overfitting), but here let us assume, that one has lot enough data, and the training and test set are organized in a such way, that good score on the training examples would also lead to good accuracy on the test data.

Answer 1

The short answer, is that you want the fastest way to reach the performance you expect/desire. This would mean first playing with some hyperparameters like learning rate, initialisation strategy, trying different optimizers, etc.

Having said that, if you managed to write your training process such that it is trivial to swap out the model completely for another one, you can also give that a go.

---

If you have a model that is known to work well for a given type of task, but is really under-performing (e.g. not converging or a large difference to results in the literature), then I would first check for some common issues in the training process:

• data quality: are the inputs scaled correctly? Should you add some
• data volume: if you don't have many samples, can you get more or use augmentations?
• loss function: are the outputs of the model being correctly consumed and the losses correctly computed? are you even using the right loss formulation?

This is essentially debugging your training setup.

I would convince myself as far as possible that none of these exist before building a different architecture or a radically different approach. The main reason for this, is that I assume you did some research into the problem and the type of models that are known to perform well - so if this doesn't work, what makes you think another architecture will? :)