I have trend data from many health departments in a local territory (eg. cardiology, orthopedics, etc...). These trends represent health service (visits, diagnostic, admissions) production, service value (proxy of type of service), and ratio of such service consumed by patients in our territory versus those consumed in other health providers than us, per department (eg. how much our patients have hip surgery in our hospitals versus other hospitals).

I would like to predict for each department or group of departments (all diagnostic or all surgery departments) those values for the next year. I have monthly data from the previous ten years.

Since there are many unmeasured sources of variability I can't expect to make long term predictions, by just one year is enough. Also, these trends have many change points (change of organization, personnel, technology), so I can mostly hope to predict the next year value once a new trend has started, predict for how long it will grow, reach a tipping point or going down.

I guess the model for each department could learn from previous growth and fall dynamics from his own past trend data and from the trends of other departments with similar dynamics and adjust for month related activity level, plus from information on where the service was provided.

Which model should I use for such a problem? I thought about a multilevel model for time series, but I guess there are more raffinate solutions.


1 Answer 1


One approach could be to simply use standard time series models, like ETS/TBATS/ARIMA for each department. These methods will, with proper care, take into consideration specific trends you might see (as well as seasonality) albeit predicting when a trend will fall/how long it will last is an incredibly difficult task. This wouldn't be an invalid approach, but the problem here is that each time series will be fit in isolation and you seemingly have large amounts of similar time series along with valuable exogenous data, much of it that does not depend on time. The approach above would mean fitting a potentially large amount of time series models which is largely impractical and tedious (in my opinion) and in the case of ARIMA you would not be able to include variables that don't vary with time. There are multivariate extensions of ARIMA methods that could help, but if you have many different time series convergence is often a problem.

This is where I would suggest you look into regression algorithms (statistical or ML) with time (in some fashion) as a covariate. Your dataset would basically be expanded so that one row is the observed response variable at any specific month + year, for any given department. Your time covariates could be two columns; one with the year, another with the month, or simply a continuous variable like 2019 + 1/12, 2019 + 2/12, etc.

You would then use "department" as a categorical variable, along with perhaps lagged values of the response (to perhaps incorporate current trend information) or perhaps variables like "average health service production in last k month(s) by department" or "median health service production in past year over all departments = over all the data", etc. The choice of how many months to take, along with the level at which you aggregate to (over a department? over all departments? over specific departments only?) will be determined based on what you think is reasonable.

If you have time related events, you could model this with indicator variables, and also perhaps use variables like "months since this time related event has started/ended", etc. to perhaps capture lagged effects. Seasonality, if it exists, could be easily captured with indicator variables/Fourier series transforms. To obtain more trend features, you might also run another regression algorithm using the past couple months of data (from the time period for that specific observation/row of data) to get the overall current trend (at that point in time). Non-time related continuous variables (like "how much our patients have hip surgery in our hospitals versus other hospitals") is a simple join to the expanded dataset by department.

In many of these feature engineering ideas, you will end up with NA values at the start of the series which implies giving up maybe a few months of data at the beginning (I think this shouldn't be too big of a deal if you have time series' that are 120 units long). Regardless, I'd recommend using time series cross validation, using the most recent data as testing indices, and walking forward until you run out of data. In your case, each test set would be comprised of 12 months worth of data, and you would add one month of data at a time to the training set (moving the test sets time period one month ahead).

I hope this helps.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.