I wonder if a binary classification problem which will steer a business process really is binary. As you see steering means affecting the outcome of the business process. As such ground truth data will be different than it would have been without the model.

What strategies are there to deal with this problem (besides a hold out group). Recently I read http://ieeexplore.ieee.org/document/7280527/ wich additionally incorporates early feedback.

Is it suitable to assume that after initial launch of such a model which predicts 0 and 1 class labels future ground truth data wich is returned with a delay of up to 30 days will in reality be up to 6 classes:

  • 0 prediction of label 0
  • 1 prediction of label 1
  • feedback that label was actually
    • 0 for a prediction of 0 (regular outcome)
    • 1 for a prediction of 1 (regular /desired outcome)
    • 0 for a prediction of 1 (model did falsely classify, groundtruth affected)
    • 1 for a prediction of 0 (model did not detect classification, suboptimal and lost some money but ground-truth considered not affected)

How could the feedback be incorporated? Would you suggest to expand the binary classification to a multi-class classification?

  • $\begingroup$ This is a really interesting question. Could maybe do with a title that summarises your general concern, as opposed to your thinking about the classifier output (which might be useful, but it's already working towards a proposed solution). I think it is less to do with specific structure of the classification, and more to with general case of interventions driven by a predictive model, and how that could affect both the model and population going forward. $\endgroup$ Feb 15, 2017 at 8:12
  • $\begingroup$ Good idea. I changed the title. $\endgroup$ Feb 15, 2017 at 8:15
  • $\begingroup$ I think reject inferencing is a good keyboard to look for stats.stackexchange.com/questions/13533/… $\endgroup$ May 20, 2017 at 9:31

1 Answer 1


One option is reinforcement learning (RL).

RL frames the problem as sequential decision-making under uncertainty. There is an agent making decisions and after the decision is made, the agent collects new data. The next round of decisions uses the new data.

What you are describing is a scenario where the distribution of "ground-truth" changes over time. Many statistical and machine learning models assume a stationary process where the underlying distribution does not change over time. There are RL algorithm that do not assume a stationary process.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.