Custom loss function in python

Question

I am trying to implement a custom loss function inspired by https://arxiv.org/pdf/2305.10464.pdf. That is:

$ L(\mathbf{x}) = (1-y) \left\lVert \mathbf{x_{true} - \mathbf{x_{pred}}} \right\rVert^2 + y \left\lVert \mathbf{1} - \mathbf{x_{true} - \mathbf{x_{pred}}} \right\rVert^2 $

The idea is that the model should reconstruct normal data as closely as possible while reconstructing anomalous data poorly. Thus, the model aims to put anomalous data outside of the domain description of the normal data. The labels y are purely to help guide the model and not to be predicted (y=1 --> anomaly , y = 0 --> normal ).

Although I find the loss function to be intuitive when comparing just two data points, I feel a bit confused when implementing this in python using keras. My understanding is that the loss function is evaluated per batch (is this really the case?). For the classical MSE loss, this would mean averaging the MSE-loss for all data points within the batch (https://keras.io/api/losses/regression_losses/). With the custom loss function, I am worried that the evaluation of the loss function won't capture the "anomaly-part" of the loss function as this would be lost in the aggregation? I'm unsure how to guarantee that these subtleties are effectively captured during model training. Any insights or suggestions on this matter would be highly valued.

Answer 1

worried that the evaluation of the loss function won't capture the "anomaly-part" of the loss function as this would be lost in the aggregation?

You're talking about weighting the minority class, the anomalies. If they have low weight, the model will focus on reconstruction error and may ignore the anomaly class label.

The OP mentions Angiulli et al.'s eqn (2.), with $\lambda$ set to $1$, and $F(\mathbf{x}) = \mathbf{1} - \mathbf{x}$.

Your choice of setting the hyperparameter $\lambda$ to unity suggests you believe there is no strong imbalance of classes in your dataset. But the notion of "anomaly", and the worry you gave voice to, suggest you'd be happier with a larger factor there. In §4.2 the authors recommend using log-spaced factors s.t. $0.1 < \lambda(\alpha) < 2$.