Evaluating recommendations quality and accuracy

Question

I'm developing a recommendation system, that should provide my clients what actions they should take in order to hit certain targets.
The underlying mechanics of the process is physical - where both actions and outcomes can be measured physically.

My current algorithm is based on a predictive model that can predict expected result for a given action. This result can be scored, and thus scoring* many outcomes can lead to finding the best action - the action that will be recommended.

*the scoring is based on distance between outcomes and targets.

My predictive model performance is good(low MAE). Its performance estimation is based on predicting already taken(historic) action and comparing its prediction to actual(historic) outcomes.

My problem arise when I try estimating the recommendations quality, since many of the recommendations are actions that were not taken, I have little ways evaluating their quality.

Ive tried using actual outcomes as 'targets' to my recommendation system, and compare the recommended action to actually taken action. I am not sure this is the right way to go, and any advice will be appreciated.

Answer 1

Unraveling the causality involved in this type of recommender system is complicated, but tractable.

Given that you have a good experimental methodology, and your evaluation numbers are generated on numbers not used for training (among other concerns), it sounds like you have the best estimate that you can get (or are in the neighborhood) using past data.

The next step would be to form a new experiment, in which you (ideally) randomly provide the recommendation for one group and not provide it for another group, which otherwise are the same. You then measure the outcomes for both groups and compare them statistically to determine the presence of an effect and its size.

Proposing that study is your next step if you're in an organization, or performing it if you have the agency to do it on your own.

Answer 2

A general approach for this is Reinforcement learning. It solves the problem for evaluating unexplored actions by choosing them stochastically, the exploration rate can be a parameter.

Another general approach is to develop a good model of the environment. Then the results on unexplored actions can be evaluated by simulation.