I wonder if a binary classification problem which will steer a business process really is binary. As you see steering means affecting the outcome of the business process. As such ground truth data will be different than it would have been without the model.
What strategies are there to deal with this problem (besides a hold out group). Recently I read http://ieeexplore.ieee.org/document/7280527/ wich additionally incorporates early feedback.
Is it suitable to assume that after initial launch of such a model which predicts 0 and 1 class labels future ground truth data wich is returned with a delay of up to 30 days will in reality be up to 6 classes:
- 0 prediction of label 0
- 1 prediction of label 1
- feedback that label was actually
- 0 for a prediction of 0 (regular outcome)
- 1 for a prediction of 1 (regular /desired outcome)
- 0 for a prediction of 1 (model did falsely classify, groundtruth affected)
- 1 for a prediction of 0 (model did not detect classification, suboptimal and lost some money but ground-truth considered not affected)
How could the feedback be incorporated? Would you suggest to expand the binary classification to a multi-class classification?