2
$\begingroup$

I have a regression problem I'm trying to build a model for: Predicting sales per person (>= 0) depending on some variables. I'm running different model types and gave deep neural networks a try. The loss functions I'm using are mean squared error and mean absolute error (or sometimes a mix).

I often run into this issue though, that despite mse and mae are being optimized, I end up with a very strong bias in the prediction, e.g. sum(training_all_predictions) / sum(training_all_real) = 0.76.


Looking at this from a small example point of view, I can't blame the model:

real <- c(10, 30, 100)
pred1 <- c(4, 14, 122)
pred2 <- c(16, 46, 122)

## mean absolute error
mean(abs(pred1 - real))
# 14.66667
mean(abs(pred2 - real))
# 14.66667

## mean squared error
mean((pred1 - real)^2)
# 258.6667
mean((pred2 - real)^2)
# 258.6667

So from a model loss point of view, these are identical solutions. However, if I were to sum up multiple predictions, I would clearly prefer pred1:

sum(pred2) / sum(real)
# 1.314286
sum(pred1) / sum(real)
# 1

So if I take the whole example, pred2 is off by 31%, while pred1 nails it. On a individual level both predictions are equal.

All other common regression loss functions I found struggle from the same problem. (Using Keras: https://keras.io/api/losses/)

Questions:

  1. Can I solve this with a custom loss functions?
    • I tried (cumsum(y_pred) - cumsum(y_test))^2 but although I got a decline of this loss over epochs, I was even further off (~0.6).
  2. Am I attacking my problem from the wrong angle?
    • I could try to build a model on cohorts, but this just feels very off, as I would have to aggregate information and would introduce cohort size as another variable.
    • Multiplying everything with a factor also sounds off, as this will likely heavily increase mse / mae again.

Edit: Specified why pred1 is better than pred2.

Edit2: Removed the reference to Estimator bias to avoid confusion.

Edit3: Increased the numbers in the example to make it more obvious.

$\endgroup$
7
  • $\begingroup$ I am not sure I understand what the problem is. Bias of an estimator is an expected value not instance value. For example I am not sure why pred1 is preferable over pred2, is it supposed to sum to some constant? If so, state this constraint clearly. $\endgroup$
    – Nikos M.
    Commented May 31, 2022 at 13:33
  • $\begingroup$ In other words state all the constraints that should be satisfied clearly $\endgroup$
    – Nikos M.
    Commented May 31, 2022 at 13:35
  • $\begingroup$ > Bias of an estimator is an expected value not instance value. You are right about that, I shouldn't mix that up. My problem is the model being off significantly when summing up multiple predictions. <hr> >In other words state all the constraints that should be satisfied clearly Thank you, I edited the question to include why pred1 is better than pred2. $\endgroup$
    – JanS
    Commented May 31, 2022 at 17:46
  • $\begingroup$ Honestly I still fail to understand what is the required condition that has to be met. For example rounding the predictions both pred1 and pred2 are identical. Please clarify what do you mean by summing and what the expected result should be. $\endgroup$
    – Nikos M.
    Commented May 31, 2022 at 18:05
  • $\begingroup$ Maybe what you really need is rounding the predictions to some fixed accuracy instead of different loss functions $\endgroup$
    – Nikos M.
    Commented May 31, 2022 at 18:10

1 Answer 1

1
$\begingroup$

Thank you @Nikos M. for your suggestions. I was about to use your post-applied factor but then gave it another try. And found what caused this. It was that the final layer was using a softplus activation function. It sounded like a perfect fit to me for this regression problem, as I only had positive valued outcomes. However this seems to cause some troubles for my DNN, which I don't understand why. Anyway, that's a different topic. Using relu in the final layer gave me much better results and also made my initial problem here disappear.

So problem is solved. I would say the answer to my question is: If you see something so far off, you shouldn't look for a way to force this. You should debug your model instead.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.