I have a neural network (MLP) that is consistently underestimating the target variable on the validation set, test set, and on the training set (by about the same amount as on the validation set and test set). In other words, the sum of the entries in the neural network regression output column (Y_hat) is something like 10% less than the sum of the entries in the target variable column (Y). The entries in the target column are greater than or equal to zero, with a decent number of zero entries. The right-side tail of the distribution of the target variable is long.
Fitting the neural network using many random seeds consistently leads to similar results (the neural networks are all biased in the same direction and by a similar amount).
The problem seems to be fairly robust to changes in important hyperparameters, including: - early stopping - learning rate schedule - model complexity - regularization (dropout, batch norm)
The problem is less severe for low-capacity neural networks.
Does anyone have any ideas as to why this persistent underestimation might be happening?