0
$\begingroup$

My problem statement : Time Series forecasting(Month wise data), training on 96 months of data and predicting next 12 months with a 3 months empty window in between.

Example :

Batch 1

***Training Data index*** <2010-01-01 -----------------------------2017-12-01>

***Unused Month Window*** <2018-01-01---2018-03-01>
***Test Month*** <2018-04-01> [Trained model with Batch 1 training data 
                               can ONLY be used for predicting this month, 
                               not any other]

Batch 2

***Training Data index*** <2010-01-01 -----------------------------2018-01-01>

***Unused Month Window*** <2018-02-01---2018-04-01>
***Test Month*** <2018-05-01> [Trained model with Batch 1 training data 
                               can ONLY be used for predicting this month, 
                               not any other]

and so on till Batch 12...

I am training 12 XGBoost models to get predictions for each of the 12 months of FY 18, hence getting 12 different feature importances against each model for the predictors used. But i want to report the feature importance of entire FY 18, instead of giving 12 different set of feature importances against each month. How would I approach that??

Evaluating a single model on the entire test dataset is not an option.

Any help is appreciated. Thanks.

$\endgroup$
2
$\begingroup$

I suspect you may be confusing the terms "test set" and "training set". Normally, the model is trained on the training set, and evaluated on the test set. The feature importance is independent of the test set, it's a property of the model that you trained.

The obvious answer is to combine all months into a single training dataset, and train a model on that, but you say it’s not an option.

Then I think your only option is to combine the results from individual months, for example by summing or averaging them and presenting the final ranking. Whether this is valid depends on the criteria used to train the trees (how the splits are determined). If there is a valid way to combine them, I still recommend against it, because I suspect it will obscure reality and discussion over the methods may distract from the result.

So I would sidestep the problem by presenting a graph like is sometimes shown for sports leagues or polls before elections. On the horizontal axis are the months, on the vertical axis the rankings. This way you visualize all info easy to grasp and you also show any trends and seasonal effects.

Example voz

$\endgroup$
  • $\begingroup$ Hey paul. Thanks for your answer. I am training 12 XGBoost models to get predictions for each of the 12 months of FY 18, hence getting 12 different feature importances against each model. But i want to report the feature importance of entire FY 18, instead of giving 12 different set of feature importances against each month. How would I approach that?? I hope I have clarified my question. :) I have also updated my question to clarify it further. $\endgroup$ – Dravidian Aug 25 at 12:36
  • $\begingroup$ Yes, I get it. My first question would be, why? A more interesting presentation could be a viz like what is sometimes shown for sports leagues, with the ranking as a function of match day (instead of teams, you would show features). That way you can see trends and seasonal effects. You can also train on all of FY18 and get the importance from the resulting model, which is straightforward. I don’t recommend to combine the importances from all months (by summing or averaging), because the info you present becomes difficult to understand and it depends on how the trees are trained. $\endgroup$ – Paul Aug 25 at 12:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.