Just one comment on regularization, as you use it in the same sentence together with normalization.
In my opition, normalization is something that should be considered very early in model evaluation and as stated by the other posters here, it heavily depends on the ML model you use (and of course your data) , how big the influence of normalization is. So for decision tree based models, it probably has little or no influence, while for KNN it most likely has a very big influence.
Regularisation on the contrary is something I would do as one of the very last steps. I would try to optimize all other parameters first (roughly), then do regularization and after that, if required I would see if I still can fine-tune some other parameters after that.
Imho regularization should not be one of the first steps, also because according my experience the regularization value has a strong interdependence with almost every other parameter. And that is not the case for all parameters. E.g. if you use
lightgbm the parameter for the maximum number of leaves the model can generate, seems to be pretty stable. If you once have optimized it for your ML problem, you can almost certainly leave it unchanged, even if you change other parameters.
So if you can identify such "stable" parameters for your model, it is a good idea to optimize them first.
On the other hand, if regularization influences almost every other parameter, you should probably do this at last to see, how far you can get without regularization and then see, if you can improve that further by applying regularization.
And of course, there are also clear signs, when regularization is required. You just have to look at the validation and train errors and how both progress. Roughly if your trainig errors decrease while your validation errors increase after doing a change, your change increased the overfitting of your model. So if you discover a large gap between training and validation error, then you should check, if you can reduce overfitting and for this regularization is just one possibility.