While designing the architecture of a Neural Network, should I consider adding regularization (like Dropout, L1/L2, etc.) even after optimizing the problem using Hyperparameter Tuning? What should be the main focus (tuning or regularization) to achieve more generalization, considering that both of these simplify the problem?
P.S.: Links to some recent publications would be of great help.
Yes, it is generally recommended to use regularization techniques, such as Dropout, L1/L2 regularization, etc., even after optimizing the problem using hyperparameter tuning. Regularization techniques help to prevent overfitting by adding constraints to the model and simplifying the model complexity, which can improve the generalization performance.
On the other hand, hyperparameter tuning optimizes the hyperparameters of the model to find the best configuration that minimizes the error on the training set.
In summary, both hyperparameter tuning and regularization are important for achieving more generalization in a neural network. Hyperparameter tuning helps to find the optimal values for the hyperparameters of the model, such as the learning rate, the number of hidden layers, etc., which can improve the performance of the model. Regularization techniques, on the other hand, help to reduce the complexity of the model and prevent overfitting, which can also improve the generalization performance of the model.
Therefore, it is recommended to use both hyperparameter tuning and regularization techniques to achieve more generalization in a neural network.
Here are some recent publications that discuss the use of regularization and hyperparameter tuning for Neural Networks:
Understanding Regularization in Deep Learning by B. Zoph and Q. V. Le
Hyperparameter Optimization for Neural Networks: A Survey by A. K. Agarwal, S. R. M. Prasad, and A. R. Menon
Dropout: A Simple Way to Prevent Neural Networks from Overfitting by N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov
Early Stopping - But When? by Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner
Reducing Overfitting In Neural Networks by Constraining the Complexity of the Model by Yarin Gal and Zoubin Ghahramani
An Overview of Regularization Techniques in Deep Learning by Danilo Jimenez Rezende and Shakir Mohamed
