1
$\begingroup$

I'm trying to find outliers in a bivariate data set. One of the input variables is the amount of time that it takes a user to do a task in our system in seconds (rounded to integer). Most interactions take on average 30-60 seconds with a large standard deviation, but it can go as high as 30 minutes, at which point the system kills the user session.

The other input is that person's speed relative to other users of the system, so a continuous variable roughly in the range of 0.5 (user takes on average about half as long as others) to about 7.x (user takes on average about 7x as long as others). So our objective is to find outliers that would indicate "it took too long to do this thing given the user's relative speed," from which we will conclude that the user was likely interrupted in the process and treat those values differently.

I ran each variable distribution through Python's distfit to try to figure out what type of distribution they follow. We had assumed a gamma on the first variable, but distfit tells me that its raw form is best fit by an alpha distribution, while the ln of it is fit by an Inverse Gaussian. It tells me that the second variable in raw form is best fit by a Johnson Su and the ln is also best fit by an Inverse Gaussian. On visual inspection, the natural logs definitely come much closer to normal distributions than the raw form, which I think might be useful.

So that's cool and all, but I don't really know where to start with fitting a bivariate distribution. We assume that the end result will be something like a scatterplot with "time to completion" on the x-axis and "relative speed rating" on the y-axis and a threshold beyond which are outliers. The threshold would be closer to 0 for faster users and further from 0 for slower users.

Is there something like distfit that would help us zero in on finding a good fit of the distribution given that neither input is normally distributed? Any idea what would be a good starting point for approaching this problem? Honestly, I'm out on a limb past the limits of my statistics knowledge at this point.

$\endgroup$

1 Answer 1

2
$\begingroup$

First, I would actually plot a two-dimensional scatter plot, just for a visual check. Next, I would suggest a non-parametric approach, a DBSCAN, or, if your clusters of "typical" users have a varying density a HDBSCAN.

enter image description here

Both algorithms are designed to look for clustered points in the data, and are especially usefull for a low-dimensional data. (H)DBSCAN will assign each point in your dataset either a cluster label, or a label of a noise, an "outlier" (gray points on the picture above), which is exactly what you want.

An alternative approach is an IsolationForest algorithm, it is designed specifically for anomalies detection problems. All three algorithms are implemented in the scikit-learn.

$\endgroup$
1
  • 1
    $\begingroup$ Excellent! Thank you, I will play with those today. Appreciate your help. $\endgroup$
    – skustes
    Oct 20, 2023 at 14:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.