2
$\begingroup$

I'm involved in a ship transit time prediction project. Is about prediction the time that a ship's cargo takes to go from port A to B in order to contract the fastest carrier company and tell the client how long it going to take, because a several times a the carrier company takes more or less (most of the times more).

We have historical database with predictive variables and stuff, the point is if the real transit time of cargo is 10 days is worse to the model estimate 9 days than eleven days, even that the absolute error is 1.

So a need a custom metric to punish positive errors more than negative ones and punish absolute large errors as well, calling error = actual - predicted.

I thought in using a weighted mean squared error, but I don't know how to balance the weights. Do you guys may help me? Do you know some other metrics that meet that requirements?

Thanks very much!

$\endgroup$
1
  • $\begingroup$ Does the business have a cost associated with each day late and a different cost with each day early and a revenue associated with on-time? If you can put this problem into the business measurements then that would be a good metric. Every day late costs $1,000, every day early costs $500 on-time revenue is $750. Then sum. I like to put metrics into how the model will be used and measured, where possible. $\endgroup$ – Craig Dec 10 '20 at 12:18
2
$\begingroup$

How about using the difference of an exponential and a linear or a polynomial functions?

For example, if $x = actual - predicted$, then $$L = w_1 (e^x-1) - w_2 x$$ The cost of error increases faster for positive values than for negative ones, and is a differentiable function of $x$. You can tweak $w_1$ and $w_2$ depending on your typical errors and your learning algorithm. This is an example, for $w_1 = w_2 = 20$:

enter image description here

$\endgroup$
1
$\begingroup$

Another straightforward approach is to use the Pinball Loss function, which you can use to predict quantiles. Let $τ$ denote the target quantile, $y$ the target and the $z$ the quantile target, the pinball loss is defined as: $L_τ= \begin{cases} (y-z)τ,\text{if}\ y\geq z \\ (z-y)(1-τ), \text{if}\ y<z\\ \end{cases}$

This is the standard loss for quantile regression and for $τ=0.50$ this is equivalent to the standard $\ell_1$ loss, i.e., prediction the conditional median value.

The only thing that remains is how to find the optimal target quantile. This depends on the business application at hand. Assuming a cost for under prediction $c_{down}$ and a cost for over prediction $c_{over}$, the optimal quantile is defined as: $τ^*=\frac{c_{down}}{c_{down}+c_{over}}$

$\endgroup$
2
  • $\begingroup$ In this approach you're supposing that is target variable is a quantile? How this works? $\endgroup$ – Allan Araujo Dec 14 '20 at 13:32
  • $\begingroup$ This is similar to the "newsvendor" problem, when you have uncertain demand of a perishable product with uneven costs. Let's assume that over-forecasting is 3-times worse than under-forecasting. The pinball loss function is similar to the $\ell_{1}$ loss, but on the one side (negative errors) would be 3-times steeper than the other. If you train your model to minimize the pinball loss, the optimal value is the $\frac{1}{1+3}$ predictive quantile. The $0.25$ quantile would actually be the better value to tell to the client, in order to minimize costs. $\endgroup$ – Akylas Stratigakos Dec 14 '20 at 18:52
0
$\begingroup$

if you are fine with non-differentiable cost functions (or more precisely non-smooth) there is a simple and intuitive metric to use involving the step function $\theta(x)$ defined as:

$$\theta(x)={\begin{cases}1&{\text{if }}x \ge 0\\0&{\text{if }}x < 0\\\end{cases}}$$

The cost defined with respect to the step function would be:

$$L(\Delta t) = w \times \theta(\Delta t) \times \Delta t$$

and it penalises only positive time differences $\Delta t$ while it assignes zero cost to negative time differences ($w$ is a free-parameter weight that can be assigned to any value, eg $1$).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.