2
$\begingroup$

I have a neural network (MLP) that is consistently underestimating the target variable on the validation set, test set, and on the training set (by about the same amount as on the validation set and test set). In other words, the sum of the entries in the neural network regression output column (Y_hat) is something like 10% less than the sum of the entries in the target variable column (Y). The entries in the target column are greater than or equal to zero, with a decent number of zero entries. The right-side tail of the distribution of the target variable is long.

Fitting the neural network using many random seeds consistently leads to similar results (the neural networks are all biased in the same direction and by a similar amount).

The problem seems to be fairly robust to changes in important hyperparameters, including: - early stopping - learning rate schedule - model complexity - regularization (dropout, batch norm)

The problem is less severe for low-capacity neural networks.

Does anyone have any ideas as to why this persistent underestimation might be happening?

$\endgroup$
2
  • $\begingroup$ What do you mean by 'sum of targets'? And the problem you are referring to can have tons of reasons to be there. Be more specific please! $\endgroup$
    – naive
    Sep 15 '18 at 17:56
  • $\begingroup$ Let me edit the original question. Thanks for the feedback. Edited. $\endgroup$
    – Solver
    Sep 15 '18 at 17:56
2
$\begingroup$

Consistently underestimating target could be due to the distribution of the target variable. If the target distribution has a negative-skew (i.e., a long tail towards lower values), then the neural network is just pattern matching to minimize those errors. For example, if the network has a squared loss function then large estimation errors in the tail are weighted more heavily than small estimation errors in the head, even though there are fewer data points in the tail. The model is trying to minimize the overall loss function and is "cheating to win" by consistently underestimating.

$\endgroup$
1
  • $\begingroup$ The target variable is non-negative with a long tail on the high end. I think the variance of the errors should basically scale with the target variable, but I am using OLS to fit the model. But I thought that OLS produces unbiased regression models even if the errors are heteroscedastic ...? $\endgroup$
    – Solver
    Sep 15 '18 at 18:30
1
$\begingroup$

I had the same issue while trying to predict LTV of players installing a free-2-play game and realised after trying all the different architectures, weights, different loss functions to optimise, etc that DNNs are not the best solution for such a problem and consistently underestimate as they are just trying to fit various continuous functions.

(Trying different loss functions like mean absolute error, mean squared logarithmic error, etc made it even worse)

Finally I went with a RandomForest Regressor and gave more training data for the tails to ensure that my Random Forest learns better on the tails, since I knew that tails is where the problem was severe. This solution worked like a charm, of course after a bit of fine-tuning.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.