Statistic methods to compute the average of a list of metric results

Question

I have a Machine Learning model that fits and predicts many time series at once, so, for each time series I have a metric result, for example, MAE. What I need is to generate an unique value for that metric for the whole model, not just for the individual predictions.

The metric results are a list, so they look like this:

metric_results = [0.2, 2.5, 4.5, 0.9, 1.6]

Possibilities tried for my data:

Mean: that doesn't look the best way to compute that, as the results don't represent the quality of the model as a whole, because as there are some zeroed values, the mean goes down, even though I know the model ain't performing that great.

Median: even worse results than mean.

Geometric mean: cannot be used because there are 0 values.

Harmonic mean: same problem as geometric mean.

So, is there a better way to compute the average of a list of these values?

Answer 1

In general when people use some kind of a ensemble a mean of the the multiple metrics is considered the 'typical' predictive capability of that model, unless there is some sort of uneven weighting of the constituent models.

So, yes, I understand that this means with a few 0 averaged in there you get a big shift in the mean. However, short of modeling mistakes or some kind of bias in building the model, that is likely not unexpected.

So, if that value is unacceptable (or not translating into a useful prediction), then you need to reconsider this particular modeling path.

Potentially try to build a single more predictive model that provides a MAE you find acceptable across the full range of your data