Let's say I have a dataset with the following format:

  • customerid
  • product
  • orders_in_last7days
  • orders_in_last6days
  • orders_in_last5days
  • orders_in_last4days
  • orders_in_last3days
  • orders_in_last2days
  • orders_in_last1days
  • orders_currentday

This dataset could have multiple customers and some customers could place $n$ numbers of orders on different days. How can I flag customers that have unusual number of purchases on the current day, by looking at the distribution of orders on the previous day for that specific customer?


The definition of an outlier per si is already quite dubious, but we can define them as those values that surpass the limit of +- 1.5 * IQR. In this case, either the standard deviation method or the Tukey method are valid options. We just need to try and see which gives better results.

# Tukey Method

n = 2 #In this case, we considered outliers as rows that have at least two outlied numerical values. The optimal value for this parameter can be later determined through the cross-validation
indexes = []

for col in df.columns[0:14]:
    Q1 = np.percentile(df[col], 25)
    Q3 = np.percentile(df[col],75)
    IQR = Q3 - Q1

    limit = 1.5 * IQR

    list_outliers = df[(df[col] < Q1 - limit) | (df[col] > Q3 + limit )].index # Determine a list of indices of outliers for feature col

    indexes.extend(list_outliers) # append the found outlier indices for col to the list of outlier indices

indexes = Counter(indexes)
multiple_outliers = list( k for k, v in indexes.items() if v > n )

Once you detect the outliers, you can either remove them or replace them with max/min limit values. In the first case, you just need to do this:

df.drop(multiple_outliers, axis = 0)

df = df.drop(multiple_outliers, axis = 0).reset_index(drop=True)

But for the second case, you should do this:

#Setting the min/max to outliers using standard deviation
for col in df.columns[0:14]:
    factor = 3 #The optimal value for this parameter can be later determined though the cross-validation
    upper_lim = df[col].mean () + df[col].std () * factor
    lower_lim = df[col].mean () - df[col].std () * factor

    df = df[(df[col] < upper_lim) & (df[col] > lower_lim)]

Finally, you can also use Isolation Forest or LocalOutlierFactor (more appropriate for Anomaly/Fraud Detection Problems).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.