How to find feature importance with multiple XGBoost models

Question

My problem statement : Time Series forecasting(Month wise data), training on 96 months of data and predicting next 12 months with a 3 months empty window in between.

Example :

Batch 1

***Training Data index*** <2010-01-01 -----------------------------2017-12-01>

***Unused Month Window*** <2018-01-01---2018-03-01>
***Test Month*** <2018-04-01> [Trained model with Batch 1 training data 
                               can ONLY be used for predicting this month, 
                               not any other]

Batch 2

***Training Data index*** <2010-01-01 -----------------------------2018-01-01>

***Unused Month Window*** <2018-02-01---2018-04-01>
***Test Month*** <2018-05-01> [Trained model with Batch 1 training data 
                               can ONLY be used for predicting this month, 
                               not any other]

and so on till Batch 12...

I am training 12 XGBoost models to get predictions for each of the 12 months of FY 18, hence getting 12 different feature importances against each model for the predictors used. But i want to report the feature importance of entire FY 18, instead of giving 12 different set of feature importances against each month. How would I approach that??

Evaluating a single model on the entire test dataset is not an option.

Any help is appreciated. Thanks.

Answer 1

I suspect you may be confusing the terms "test set" and "training set". Normally, the model is trained on the training set, and evaluated on the test set. The feature importance is independent of the test set, it's a property of the model that you trained.

The obvious answer is to combine all months into a single training dataset, and train a model on that, but you say it’s not an option.

Then I think your only option is to combine the results from individual months, for example by summing or averaging them and presenting the final ranking. Whether this is valid depends on the criteria used to train the trees (how the splits are determined). If there is a valid way to combine them, I still recommend against it, because I suspect it will obscure reality and discussion over the methods may distract from the result.

So I would sidestep the problem by presenting a graph like is sometimes shown for sports leagues or polls before elections. On the horizontal axis are the months, on the vertical axis the rankings. This way you visualize all info easy to grasp and you also show any trends and seasonal effects.