How important is the balance between the components of a loss value?

I'm training a neural network for predicting the location of a single object, so the prediction consists of 4 values: x, y, width and height. Without weight decay, the loss starts at around 0.3. If I add weight decay with a factor of 0.004, the loss jumps to around 22. Does this mean the network will give more importance to minimizing the weights over finding good coordinates? How important is the balance between the components of a loss function?