Finding Outliers in Resource Forecast Data

Question

Unsure if this is the correct place to place this, please close if so.

I'm a workforce analyst at a large retail company, I own and maintain all the forecasting for our retail stores. This is based on Product Sales Forecast which I run through a model which has various tasks (such as changing items on a shelve or selling a car alarm for example) binned into timed categories, e.g

Item A takes 30 seconds, so 500 product forecasts will allow you (0.5 * 500) or 4.1 hours which are then further binned into Weekly allocations.

Now, as, for each iteration of this model I've run, there are always outliers (read mistakes), such as Store Y has no product volume to Product Group X in Week 22, which is a mistake from the finance team.

My senior has said this has always been the case for the 8 or so years he's been with the company.

Now, in my head, I assume I would be able to find outliers based on historical data using some sort of statistical method, however, I have no idea where to start, my data is as follows and is around 10 million rows of data.

import pandas as pd
import numpy as np

data =  [19,21,24,18,3]
pg = ['PG','ZF','AA','GG','ZF']
location = ['AA_1','AA_1','AA_2','AA_2','AA_2']
weeks = [1,1,2,2,2]

df = pd.DataFrame({'Location' : location,
            'productGroup' : pg,
            'Week' : weeks,
            'productVolumes' : data })
print(df)


  Location  productGroup    Week    productVolumes
0   AA_1    PG  1   19
1   AA_1    ZF  1   21
2   AA_2    AA  2   24
3   AA_2    GG  2   18
4   AA_2    ZF  2   3

in your expert opinions, what would be the best method to use?

Answer 1

There is not one way of detecting outliers. Most statistical methods assume an underlying normal distribution. If this is the case you could apply for example an Thompson Tau test. You can also define some own criteria. In your use case I think an important question is: what will happen to the quality of the model if you drop or modify missing data. This gives you an idea how to handle these missing data points.