# Univariate Outlier Detection

Let's say I have a dataset with the following format:

• customerid
• product
• orders_in_last7days
• orders_in_last6days
• orders_in_last5days
• orders_in_last4days
• orders_in_last3days
• orders_in_last2days
• orders_in_last1days
• orders_currentday

This dataset could have multiple customers and some customers could place $$n$$ numbers of orders on different days. How can I flag customers that have unusual number of purchases on the current day, by looking at the distribution of orders on the previous day for that specific customer?

The definition of an outlier per si is already quite dubious, but we can define them as those values that surpass the limit of +- 1.5 * IQR. In this case, either the standard deviation method or the Tukey method are valid options. We just need to try and see which gives better results.

# Tukey Method

n = 2 #In this case, we considered outliers as rows that have at least two outlied numerical values. The optimal value for this parameter can be later determined through the cross-validation
indexes = []

for col in df.columns[0:14]:
Q1 = np.percentile(df[col], 25)
Q3 = np.percentile(df[col],75)
IQR = Q3 - Q1

limit = 1.5 * IQR

list_outliers = df[(df[col] < Q1 - limit) | (df[col] > Q3 + limit )].index # Determine a list of indices of outliers for feature col

indexes.extend(list_outliers) # append the found outlier indices for col to the list of outlier indices

indexes = Counter(indexes)
multiple_outliers = list( k for k, v in indexes.items() if v > n )


Once you detect the outliers, you can either remove them or replace them with max/min limit values. In the first case, you just need to do this:

df.drop(multiple_outliers, axis = 0)

df = df.drop(multiple_outliers, axis = 0).reset_index(drop=True)


But for the second case, you should do this:

#Setting the min/max to outliers using standard deviation
for col in df.columns[0:14]:
factor = 3 #The optimal value for this parameter can be later determined though the cross-validation
upper_lim = df[col].mean () + df[col].std () * factor
lower_lim = df[col].mean () - df[col].std () * factor

df = df[(df[col] < upper_lim) & (df[col] > lower_lim)]


Finally, you can also use Isolation Forest or LocalOutlierFactor (more appropriate for Anomaly/Fraud Detection Problems).

This problem requires you to distinguish customer behavior . One customer’s purchase pattern may be different from another. I am assuming that you are collecting this data daily . Let’s assume that the data for each day is a vector In multi- dimensional space . you can calculate Mahalanobis distance between the vector for current day which will be of shape (1, 8) and all the history for the customer which will be of shape (n-1 , 8) .

You may want to Standardize the values in the columns since Mahalanobis distance assumes that the the data within each dimension is normally distributed .

Some of the days the customer may have not made any purchase at all .. so you may have a sparse matrix for some customers . While calculating the Mahalanobis distance you will need to calculate the inverse of the covariance matrix . This may give you an error . Try to overcome this issue by using a pseudo- inverse .