In order to validate a recommender model, a usual approach is create a hold-out set that will provide random suggestions (similar to an A/B testing setup). However, in healthcare applications, this cannot be possible as a random suggestion can put at risk a patient's life. Hence, what is a reasonable approach to validate the model?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ Could you provide a little bit more detail about what sort of work you're doing? I'm assuming a lot, like that the randomness relates to group assignment and not the type of treatment itself, but there isn't much detail here. $\endgroup$– Upper_CaseApr 8, 2019 at 21:27
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
You should still be able to use a validation set to evaluate the model, whether or not you pursue an experimental approach. (Specific features of your model and investigations may tweak these, but this is based on what's already been posted alone).
There is nothing wrong with A/B group assignment and testing in a medical context, with a few caveats (this list is not exhaustive):
- The relevant clinical/medical knowledge must be in a state of equipoise (it's not already clear that one approach is better than another, or which is better is genuinely not known).
- Individuals should be aware that they are participating in a study, and that they are being routed to group A or B, and have the option to decline their assignment (or, conversely, they have been made aware of the experimental assignment and have consented to participate in advance).
- An institutional review board should evaluate your proposed experiment and signed off on it. This, of course, presupposes that you have access to such a board composed of members able to make those assessments.
Those can be a tall order, but you don't necessarily have to perform a prospective, double-blind experimental study in order to glean some information. A retrospective study could provide some insight as well, and your process for the validation set would be something like:
- Prepare your recommender model
- Feed your data through the model, without looking at actual outcomes
- Match your model output to actual outcomes to see whether or not people followed the recommendation (whether they ever saw that recommendation or not)
- Compare the results of people that ended up going with each recommended approach (A vs. B), as well as those who "followed" the recommendations or not (Recommended-A-did-A vs. Recommended-A-did-B, etc.)
Retrospective studies are generally not as good as well-designed, well-executed prospective experimental studies, but they can still provide a lot of information. In situations where prospective experimentation is impossible or undesirable, the information a retrospective study provides may be the very best you can actually get.
