Suppose you have a machine learning system, which aims to predict whether or not the recipient of a parcel will be at home, before delivery is attempted. For example, Alice is working from home, so the likelihood of her being at home at the time of delivery is very high. Bob, however, is working on a construction site, so he is presumed to not be at home during possible delivery windows.

The system would learn by making predictions about each person, and then receive feedback whether or not that prediction was correct. If the system predicted that the recipient would be at home, then a delivery attempt would be made, and it could be recorded whether the recipient was actually at home or not.

However, if the system would predict that the recipient was not at home, then it would not be possible to verify whether that prediction was correct. It could then be presumed that the system would always predict that the user was not at home.

Possible Solutions

I came up with two possible strategies to mitigate this issue. First, even if the system stated that the recipient was likely not at home, there would be a random chance for the delivery attempt to be made regardless. This would then be able to confirm whether or not the system was correct. This, however, would come to the downside of the added "cost" of attempting to deliver a parcel to someone who was likely not at home.

The second, somewhat more elegant solution, is to ask people whether or not they were at home during the initial delivery window, when they pick up their package. This initially seems better, but could lead to negative responses from recipients. For instance, they could be upset that they were at home, but were not delivered their parcel, because a computer didn't deem them worthy. Or, it could lead to recipients lying about having been at home, as for them, being categorized as "someone, who is likely at home" is a strictly positive categorization. All the downsides of being categorized as such only affect the delivery service, but not the recipient. It is thus in the best interest to give dishonest feedback in an attempt of being categorized more favourably.

Is there a better way of modelling this? Or would I just have to "bite the bullet" and accept that people will attempt to game the system?

  • $\begingroup$ I should note that this is merely an educational example. I'm not actually trying to build such a system for productive use. $\endgroup$
    – MechMK1
    Commented Dec 13, 2021 at 17:16
  • 1
    $\begingroup$ The solution is to assign costs to each prediction. As you state it, predicting not at home, has zero cost, so this prediction can always be made since it carries no cost, while the other prediction carries cost. However if a cost is assigned (eg delivery delay costs) then the always same prediction problem is mitigated $\endgroup$
    – Nikos M.
    Commented Dec 13, 2021 at 17:45

1 Answer 1


What you are essentially trying to do here (if I understand you correctly) is collect data (i.e. labels) while also predicting the labels.

Your second approach does look promising. One question is, why would you assume people want to lie about them being home when they are not? don't they want to get their package? - it doesn't seem like a assumption you can make, in this case, if you are trying to verify the predictions - human annotated data - in your case, manually provided yes/no feedback should always take more weight than any prediction your system would make.

After collecting the manual responses - you would want to use these responses to make your predictions better - and once you have reached certain threshold of accuracy (90%) you would stop manual response collection.

  • $\begingroup$ The reason I believe people would lie is, because there may be an off-chance they're at home. For example, they may expect an important shipment and took the day off, or are at home sick. They have nothing to lose by lying, essentially. $\endgroup$
    – MechMK1
    Commented Dec 13, 2021 at 21:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.