1
$\begingroup$

Hi I am currently testing multiple loss on my code using PyTorch, but when I stumbled on log cosh loss function I did not find any resources on the PyTorch documentation unlike Tensor flow which have as build-in function
is it excite in Pytorch with different name ?

$\endgroup$
0
$\begingroup$

Yes the pytroch is not found in pytorch but you can build on your own or you can read this GitHub which has multiple loss functions

class LogCoshLoss(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, y_t, y_prime_t):
        ey_t = y_t - y_prime_t
        return T.mean(T.log(T.cosh(ey_t + 1e-12)))
$\endgroup$
0
$\begingroup$

The accepted answer results in loss values of infinity when there is a large error between the true and predicted values. Here is a numerically stable version:

def log_cosh_loss(y_pred: torch.Tensor, y_true: torch.Tensor) -> torch.Tensor:
    def _log_cosh(x: torch.Tensor) -> torch.Tensor:
        return x + torch.nn.functional.softplus(-2. * x) - math.log(2.0)
    return torch.mean(_log_cosh(y_pred - y_true))

class LogCoshLoss(torch.nn.Module):
    def __init__(self):
        super().__init__()

    def forward(
        self, y_pred: torch.Tensor, y_true: torch.Tensor
    ) -> torch.Tensor:
        return log_cosh_loss(y_pred, y_true)

Why current answer isn't stable

The accepted answer doesn't work when the error term is very large because torch.cosh will go to infinity very quickly. For instance, here is the output of a script where I printed out the values of torch.cosh(x) and torch.log(torch.cosh(x)):

x: cosh: torch.cosh(x), log: torch.log(torch.cosh(x))
1: cosh: tensor([1.5431]), log: tensor([0.4338])
...
9: cosh: tensor([4051.5420]), log: tensor([8.3069])
...
49: cosh: tensor([9.5367e+20]), log: tensor([48.3069])
...
89: cosh: tensor([2.2448e+38]), log: tensor([88.3069])
90: cosh: tensor([inf]), log: tensor([inf])

This is running on a CPU and results will likely vary, but it shows that even for small inputs (~90), the result of T.log(T.cosh()) will go to infinity.

Also of note is that in the torch docs for cosh, they state:

When input is on the CPU, the implementation of torch.cosh may use the Sleef library, which rounds very large results to infinity or negative infinity. See here for details.

Derivation of improved implementation

I looked at the source for Tensorflow's LogCoshLoss which is numerically stable for large errors (I tested it to see).

They perform the calculation as:

def _logcosh(x):
    return x + tf.math.softplus(-2. * x) - tf.cast(
        tf.math.log(2.), x.dtype)
        
return backend.mean(_logcosh(y_pred - y_true), axis=-1)

They use softplus, which Torch also has an equivalent of in torch.nn.functional.softplus. In the docs for SoftPlus, Torch states that:

For numerical stability the implementation reverts to the linear function when $input \times \beta > threshold$.

Extra: why SoftPlus is differentiable

They are able to work around replacing values not being differentiable by writing a custom backward kernel for softplus here. Notably, for the case where the input is a scalar (the simpler case), they calculate the gradient as:

scalar_t z = std::exp(b * beta);
return (b * beta) > threshold ? a : a * z / (z + scalar_t(1.));

By computing the gradient this way, they are able to work around the operation of replacing values above the threshold not being differentiable.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.