0
$\begingroup$

I am giving a toy example for describing a real world business problem. Let's say I am a publisher and I have some book stores to visit. By visiting those stores I will check whether they have sufficient stock of my books, they are visible on shelf etc. Now, I am training a model, to recommend me the stores to visit. I have, say, 20 features and some historical data which has a target variable which represents whether a store was visited or not (1/0). I trained a RandomForestClassifier model on that data and this is the feature importance I got.

feature importance feature_14, feature_2 are more important than feature_11. Now, assume that feature_11 is a feature with high business importance. As an example, let's say feature_11 is the number of books on shelf and I want to put more importance on it in deciding whether to visit a store. Is there any way to put more importance on this feature_11 than feature_14 or feature_2? From historical data, model has learned that it is third most importance variable but I want to make it a key deciding factor. Is there any way of doing it?

$\endgroup$
0
$\begingroup$

If I am understanding the real-world aspect of this correctly, it sounds like you are trying to decide on a set of sites to visit. Overall model "variable importances" are generally not a great approach here, and you want to try to find something from the predictive model itself related to the decision-of-interest.

You may want to consider using something different from the previous round of site-visits as the target variable. Instead, the modeling target should be an expected benefit that depends on a site visit. Then, after a model is built, you can use the results to generate a ranked list site-visits predicted to provide the largest benefit.

One approach would be:

  1. Define some profitability or other benefit metric, that was available both before and after the site visits in your data were made.
  2. If your system can model on a continuous variable, use the before-after difference directly as the target. Otherwise, bin the differences into t-shirt size classes (S/M/L/XL if you can run a multi-class classifer; S/L if you are limited to binary modeling)
  3. Using site-visit as an input variable, build your random-forest model on your benefit-difference target.
  4. Now run your cases through the predicted model, once with visited=No and once with visited=Yes.
  5. Sort the results in order of biggest predicted difference between "visited" predictions. The top N sites that provide the biggest positive difference are the first N sites to visit.

If feature_11 is of business importance, i.e. it is related to profitability or some other success metric, it should fall naturally out of the analysis. However, a word of caution: For any kind of prediction based on past examples method to work, there needs to be some random variation between the variables. For example, if every case where feature_11 is large was also visited last year, then the model doesn't have anything to go on when feature_11=large/visited=No are suspect, and predictions assuming those conditions are suspect.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.