How to deal with a machine learning model which affects future ground truth data?

Question

I wonder if a binary classification problem which will steer a business process really is binary. As you see steering means affecting the outcome of the business process. As such ground truth data will be different than it would have been without the model.

What strategies are there to deal with this problem (besides a hold out group). Recently I read http://ieeexplore.ieee.org/document/7280527/ wich additionally incorporates early feedback.

Is it suitable to assume that after initial launch of such a model which predicts 0 and 1 class labels future ground truth data wich is returned with a delay of up to 30 days will in reality be up to 6 classes:

• 0 prediction of label 0
• 1 prediction of label 1
• feedback that label was actually
  • 0 for a prediction of 0 (regular outcome)
  • 1 for a prediction of 1 (regular /desired outcome)
  • 0 for a prediction of 1 (model did falsely classify, groundtruth affected)
  • 1 for a prediction of 0 (model did not detect classification, suboptimal and lost some money but ground-truth considered not affected)

How could the feedback be incorporated? Would you suggest to expand the binary classification to a multi-class classification?

Answer 1

One option is reinforcement learning (RL).

RL frames the problem as sequential decision-making under uncertainty. There is an agent making decisions and after the decision is made, the agent collects new data. The next round of decisions uses the new data.

What you are describing is a scenario where the distribution of "ground-truth" changes over time. Many statistical and machine learning models assume a stationary process where the underlying distribution does not change over time. There are RL algorithm that do not assume a stationary process.