I'm developing a recommendation system, that should provide my clients what actions they should take in order to hit certain targets.
The underlying mechanics of the process is physical - where both actions and outcomes can be measured physically.
My current algorithm is based on a predictive model that can predict expected result for a given action. This result can be scored, and thus scoring* many outcomes can lead to finding the best action - the action that will be recommended.
*the scoring is based on distance between outcomes and targets.
My predictive model performance is good(low MAE). Its performance estimation is based on predicting already taken(historic) action and comparing its prediction to actual(historic) outcomes.
My problem arise when I try estimating the recommendations quality, since many of the recommendations are actions that were not taken, I have little ways evaluating their quality.
Ive tried using actual outcomes as 'targets' to my recommendation system, and compare the recommended action to actually taken action. I am not sure this is the right way to go, and any advice will be appreciated.