I'm very new to writing cost functions for optimization and I have what may be a basic question or just a misinterpretation.

I have multiple cost functions that I'd like to add up into one total cost function. Here is a simplified example:

Say I want to maximize the bounciness $b$ of a bouncy ball while minimizing its weight $w$. The weight value goes into a function that computes the bounciness but we don't know what the function looks like.

If I define two different cost functions as follows:

$ C_1 = \frac{1}{b} $

$ C_2 = e^{w} $

And I define a total cost function as a function of both of them like:

$ C_{tot} = C_1 + C_2 $

Let's say $C_1$ varies wildly between values on the order of 10,000 and 1,000,000 while $C_2$ tends to vary between orders of 0.1 and 10 depending on the input $w$.

I want to minimize the cost of course. Is it typical to normalize these two cost functions so that they are weighted closely in the total cost function? For example so they only vary between values of 0 and 1.

Otherwise I can see a minimization algorithm glancing over variations in $C_2$ for the much larger changes found in $C_1$.

Are the weights in the cost function typically determined through trial and error or is there a straight forward method to determine what they should be?

  • $\begingroup$ It depends on what you value…so what do you value? // Also, keep in mind that your $C_1$ and $C_2$ have units, so you have to do something to allow yourself to add different units. You might just divide each quantity by the units go wind up with a unitless value. You might convert both to some common units (such as if you have one measurement that is a distance in meters and another that is also a distance but in kilometers). $\endgroup$
    – Dave
    Commented Apr 13, 2022 at 1:16
  • $\begingroup$ I thought that the formulations for $C_1$ and $C_2$ are meant to represent what I value in the total cost function. i.e. I don't value the change in low values of $w$ very much but as it gets higher I value it exponentially more. Does that make sense? Thanks for the note on units as well. Let's just consider that the two cost function values are unitless for this question. Thanks Dave. $\endgroup$ Commented Apr 13, 2022 at 20:04
  • $\begingroup$ If you take $C_{tot} = C_1+C_2$, this means that you value equally changes to $C_1$ and $C_2$. For instance, you would be perfectly willing to increase $C_1$ by $1$ if $C_2$ decreased by $2$. If that is not the case, then $C_{tot} = C_1+C_2$ is not a good choice of a cost function. $\endgroup$
    – Dave
    Commented Apr 13, 2022 at 20:07
  • $\begingroup$ Ok right that makes sense. Is there a name for this type of optimization involving multiple cost functions that add into a total cost function without a dataset to compare outputs to? I can't seem to find examples of it on the internet by just searching for "cost function optimization." All I'm finding are optimizations involving a single cost function that fits a model to a dataset. Thanks again. $\endgroup$ Commented Apr 13, 2022 at 20:22
  • $\begingroup$ That just sounds like optimization to me. You have a cost function; go optimize it. That it is formed by adding multiple other cost functions doesn't change the fact that you're optimizing $C_{tot}$. // If you want to have restrictions, like optimizing $C_{tot}$ without letter either $C_1$ or $C_2$ exceed a certain amount, that gets into constrained optimization like Lagrange multipliers and Karush–Kuhn–Tucker conditions. $\endgroup$
    – Dave
    Commented Apr 13, 2022 at 20:50


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.