Why might a neural network consistently underestimate its target?

Question

I have a neural network (MLP) that is consistently underestimating the target variable on the validation set, test set, and on the training set (by about the same amount as on the validation set and test set). In other words, the sum of the entries in the neural network regression output column (Y_hat) is something like 10% less than the sum of the entries in the target variable column (Y). The entries in the target column are greater than or equal to zero, with a decent number of zero entries. The right-side tail of the distribution of the target variable is long.

Fitting the neural network using many random seeds consistently leads to similar results (the neural networks are all biased in the same direction and by a similar amount).

The problem seems to be fairly robust to changes in important hyperparameters, including: - early stopping - learning rate schedule - model complexity - regularization (dropout, batch norm)

The problem is less severe for low-capacity neural networks.

Does anyone have any ideas as to why this persistent underestimation might be happening?

Answer 1

Consistently underestimating target could be due to the distribution of the target variable. If the target distribution has a negative-skew (i.e., a long tail towards lower values), then the neural network is just pattern matching to minimize those errors. For example, if the network has a squared loss function then large estimation errors in the tail are weighted more heavily than small estimation errors in the head, even though there are fewer data points in the tail. The model is trying to minimize the overall loss function and is "cheating to win" by consistently underestimating.

Answer 2

I had the same issue while trying to predict LTV of players installing a free-2-play game and realised after trying all the different architectures, weights, different loss functions to optimise, etc that DNNs are not the best solution for such a problem and consistently underestimate as they are just trying to fit various continuous functions.

(Trying different loss functions like mean absolute error, mean squared logarithmic error, etc made it even worse)

Finally I went with a RandomForest Regressor and gave more training data for the tails to ensure that my Random Forest learns better on the tails, since I knew that tails is where the problem was severe. This solution worked like a charm, of course after a bit of fine-tuning.

Answer 3

It seems like you are dealing with a zero inflated lognormal (ZILN) distributed target. In this case standard error loss / metrics are not correctly capturing the error structure. I suggest to take a look at https://github.com/google/lifetime_value and the corresponding paper, which has a keras implementation of the corresponding loss functions and layers. It also shows good visualization techniques for such prediction problems.

Similarly pytorch and xgboostlss offer same implementations.