I have a number of features X and a target which is the revenue generated by this interaction. More often than not no revenue is generated, but if there is it can differ considerably, it has a big right tail. I've made a model to estimate the expected revenue generated by a new x. Due to the weird distribution of the revenues I have split the model up in two parts, first estimating the probability of revenue generation and then conditionally on the fact that somebody will book, predict the amount of $log($revenue$)$. The first model is trained on a dummy variable which indicates revenue and the second one is trained on a subset of the data which contains all interactions that generated revenue.

After I have these two models P (probability of generation) and R (expected revenue generated conditioned on > 0) we can calculate the expected revenue for a new x using:

$$f(x) = P(x)e^{R(x)}$$

I've done a reasonable amount of cross validation on both the models individually and they work decent, however when they are combined they severly underestimate the amount of revenue generated. I have a good enough number of rows for the dimensionality of my data and I'm using Logistic Regression for the probability (linearity assumption is somewhat reasonable, however unbiased probabilities are very important) and Gradient Boosting for revenue estimation (got the best results for the second portion).

What should I do about the underestimation of the revenue? What are alternative unbiased probablity estimators, should I train a neural network with a hidden layer for the log-loss function? Why does the combined model underestimate? Is there a good alternative to do it in one model? Which would not increase small errors into huge errors due to the multiplication and exponent?


Why not just use quantile regressions?

Usually, in regressions, the coefficients are estimated for the average case: $\min_\beta \sum_i||X_i\cdot\beta -y_i||^2$.

If you want, you can estimate your coefficients so that they tell you things like $P(R=y)>p$. You give a probability $p$, and the model gives you $y$. You can even build probability density function out of these estimation techniques.

In quantile regressions, you solve this optimization problem for every quantile $\tau$ that you want, $\min_\beta \sum_{i|y_i\geq X_i\cdot\beta}\tau|X_i\cdot\beta-y_i| + \sum_{i|y_i< X_i\cdot\beta}(1-\tau)|X_i\cdot\beta-y_i|$.

This is a more robust approach to modeling the revenue distribution. Usually people use quantile regressions because software is readily available, but they can be implemented for other models as well. I work in an institute that does exactly that.

  • $\begingroup$ Thank you for your reply. This however does not solve my issue of not getting the expected revenue right in crossvalidation, you are trying to get me to use a different metric which is not applicable for my use case. It is interesting though (for different use cases) $\endgroup$ Jun 20 '16 at 11:59
  • $\begingroup$ One of your questions was Is there a good alternative to do it in one model?. A quantile regression combines your two models for revenue and probabilities. $\endgroup$ Jun 20 '16 at 15:01
  • $\begingroup$ I'm sorry about that comment I was a little grumpy. You might be right that this is what I need, I'll read up about it a bit more, thanks. $\endgroup$ Jun 24 '16 at 11:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.