This has been a spot of confusion for me lately upon reading more into regularisation in neural networks.
I have always viewed weights as being a measure of the importance of a feature in a model. E.g. A model of ice cream sales starts off with random weights and eventually learns to increase the weights connected to the temperature feature.
With regularisation (L2, I have been looking at most recently), small weights are favoured more by the model. This makes good sense to me when we are trying to prevent overfitting in our models. The model generalises well between different pieces of input data because the smaller weights give their associated feature less of an impact in the model. This will draw out the true patterns of the model as opposed to a few cases that may skew the weights.
Now my confusion is what if a feature like temperature is a consistent predictor of ice cream sales in our model, what does its associated weights look like after regularisation? Is their impact on the cost function so great that the model doesn't prefer smaller weights instead? Also how do folks usually set a value for lambda?