Fitting a model as a significance testing alternative

Question

My colleagues use somewhat unusual approach to estimate how far away performance of some of our company related processes diverge from historical levels. Their nature isn't really relevant for the purpose of this discussion. We may just assume here, that the final result can be represented with some real number which fluctuates over time and has some seasonality components.

They fit a regression model (with XGBoost) on historical data excluding the most recent period (say, the last month) and than compare the most recent values with model's forecast.

I argue, that this is basically an overkill and some sort of a crooked statistical significance test. And rather than fitting a model and making a forecast, we should actually calculate some normalised historical indicators (stratified by relevant groups and taking into account seasonality, where appropriate) and then do a standard significance test using a significance level, that would make sense from a business perspective.

1. Am I right or not? Are there any better ways to approach a problem of estimating performance and comparing it to historical data?
2. Does approach with modelling/forecasting has any legs? How it can be justified from a statistical standpoint?

Answer 1

Depends on the question you're asking, I suppose. You are right that you need some kind of significant test. The null hypothesis is that performance is "as expected," hasn't changed, and that any deviation from that is just sampling error or noise.

But what is "as expected"? if the business process were indeed static, it'd be straightforward. You know the distribution of measurements and can compute p-values or something.

But you have seasonality, and variation by some grouping. You want to control for that? sure. Doing that is also building some kind of time series model.

xgboost isn't particularly suited to timeseries forecasting, in the way that maybe FB prophet is. Also, you really want some distribution over predictions to assess how unusual the actual value is. prophet should give you that, but you would need to run xgboost with k-fold CV or something to start to infer the distribution of its prediction.

Like, I'm wondering what it means to compare one xgboost prediction to the actual result. What is "too different" and how would you know?

You're sort of on the same track but yes I think you are looking for a significance test at some level, and probably a better tool than xgboost.

Answer 2

It's probably a quite subjective matter, here is my opinion:

• I would say that this method makes a lot of sense if the ML model uses a number of features, something which would be hard to factor properly in a standard statistical test (especially if some kind of feature selection is involved).
• Also in favor of the method, statistical significance tests are usually interpreted in a binary way (typically p-value higher or lower than 0.05) even though the underlying p-value is continuous: there's very little different between a p-value of 0.049 and 0.51 but the two will be interpreted in a completely opposite way.
• However I agree with you that statistical significance tests are much more grounded in statistical theory, with a clear interpretation especially about levels of confidence.

Overall I can see an interest of using such a method from a practical viewpoint, but it is indeed limited in terms of theoretical interpretation and reliability.

Disclaimer: I'm not a statistician, I might be biased towards practical ML methods ;)