Getting monthly revenue predictions for outlets

Question

I am often presented with a task of predicting monthly revenues of retail outlets. Say I have a training set of N outlets, each associated with a series of historical monthly revenues (target) and a set of features which are time-independent (e.g. location (and all associated geofeatures), outlet type, trading area etc.) Then I have a test set of M outlets for which I need to predict monthly revenues for a specified time period (e.g. Q1-Q3 2019).

The usual approach I take consists of adjusting historical revenues on the training set for trend, seasonality and inflation, then take the average for each outlet and use it as a target, train my regression model (could be anything but mostly LightGBM), get predictions for the test set. These predictions are, again, averages for each outlet adjusted for trend, seasonality and inflation, so I need to deadjust them to obtain monthly predictions. So basically the scheme is: monthly historical revenues -> adjusted average as a target -> adjusted average as a prediction -> reconstructed monthly predictions (hope this makes sense).

The whole approach is pretty tedious and feels unnatural. I wonder whether there are frameworks that would allow the use of historical monthly revenues "as is" as a target and produce predictions for the specified time range.

There is, of course, panel regression, but 1) my features don't change over time so the idea seems superficial and 2) gradient boosting in panel regressions isn't exactly a hot topic so implementations are rare (if any exist).

Answer 1

LTSTM models are very good at time-series data, so look into that for starters.

Additionally I'd like to ask why you need to adjust and deadjust your model at all?

Even simpler models like boosted trees, random forest or even a plain regression should give you good predictions by simply including trends, seasons, etc. as predictors.

Edit:

I try to be a bit more comprehensive in my answer. Your problem is actually a classical problem of sales prediction, especially for retail stores like supermarkets, etc.

A great resource to see solutions to this problem can be found here:

Kaggle Walmart Sales Forecast

You will find a lot of notebooks with great approaches and solutions (e.g. based on the aforementioned LTSTM model).

But let's also try to point out a basic solution right here:

*Data** Let's assume we have data that looks basically like this:

Store  |  Month  |  Sales  |  Store_Size  |  ...
East   |  Jan    |  $0.9M  |  500         |  ...
East   |  Feb    |  $1.1M  |  500         |  ...
North  |  Jan    |  $1.5M  |  800         |  ...
North  |  Feb    |  $2.0M  |  800         |  ...

Some of the ... variables will be constant for each store (store info), some will be related to the date (seasonal) and some might vary for the store and the month (trends).

Problem

Trying to predict the sales of store West for March given all constant variables associate with that store but NOT the trend variables.

It is important to understand seasonal variables here! These are engineered features based on the month. E.g. we can easily compute the influence on sales for each month (e.g. trends like christmas sales, etc.) these are not trends because they are tied ONLY to months and not any other pattern.

Solution

Suppose we fit an XGB model to predict the sales, the simplest solution would be to simply only use features in the model that well be available at the point of prediction, so only seasonal and store info.

This might already deliver a very good model and might be enough for prediction. If you have no information on trends, you cannot add them to the model and therefore should not.

However you easily feature engineer trend data based on GAN or MICE models to augment your data.

Finally please understand: machine learning models work best if you simply throw all available data at them and do not get too cute with feature selection and engineering. XGB and similar models will identify trends by themselves and model the ideal relation between the other factors for you.

Degree of freedom, etc. really does not apply here.

Answer 2

Just a brief note on panel regression. If you look at fixed effects ("fixed" refers to each store having an own "identity"), then you can simply apply "least square dummy varaible" (LSDV) regression. LSDV is just another way to represent fixed effects. The idea is that you have a model such as:

$$y=\beta_0+\beta_1 X + \gamma I + \theta t + \lambda I_t + u.$$

Say $y$ is revenue, $X$ are time constant variables, $I$ is a set of "dummies" (each store minus one gets its own dummy indicator), $t$ is a count variable for time, and $I_t$ is an indicator for (say) month of the year. $I$ will capture store "fixed effects" and $t$ will capture things like inflation while $I_t$ will capture monthly effects.

While models like this are generally used in linear settings (e.g. OLS), they can also be employed in other settings. I would suggest looking into "generalised additive models" (GAM) which are generalisations of linear models but which are able to capture "wild" non-linear effects. Here is an R example for GAM. I think the model logic will also generalise to boosting approaches. However, in the end it is up to a try.