0
$\begingroup$

I have a Machine Learning model that fits and predicts many time series at once, so, for each time series I have a metric result, for example, MAE. What I need is to generate an unique value for that metric for the whole model, not just for the individual predictions.

The metric results are a list, so they look like this:

metric_results = [0.2, 2.5, 4.5, 0.9, 1.6]

Possibilities tried for my data:

Mean: that doesn't look the best way to compute that, as the results don't represent the quality of the model as a whole, because as there are some zeroed values, the mean goes down, even though I know the model ain't performing that great.

Median: even worse results than mean.

Geometric mean: cannot be used because there are 0 values.

Harmonic mean: same problem as geometric mean.

So, is there a better way to compute the average of a list of these values?

$\endgroup$
1
  • 1
    $\begingroup$ Could you please explain more about the task you’re trying to accomplish and why taking the mean or median of a collection of measurements makes for a poor summary of those measurements? $\endgroup$
    – Dave
    Nov 5, 2022 at 15:43

1 Answer 1

1
$\begingroup$

In general when people use some kind of a ensemble a mean of the the multiple metrics is considered the 'typical' predictive capability of that model, unless there is some sort of uneven weighting of the constituent models.

So, yes, I understand that this means with a few 0 averaged in there you get a big shift in the mean. However, short of modeling mistakes or some kind of bias in building the model, that is likely not unexpected.

So, if that value is unacceptable (or not translating into a useful prediction), then you need to reconsider this particular modeling path.

Potentially try to build a single more predictive model that provides a MAE you find acceptable across the full range of your data

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.