1
$\begingroup$

I would like to do outlier or anomaly detection on the disk free space data. Sample dataset as below (I don't have any label dataset):

date                   free_space (GB)
2019-05-15 09:00:00    102.65
2019-05-15 09:05:00    102.69
2019-05-15 09:10:00    103.11
2019-05-15 09:15:00    102.58
2019-05-15 09:20:00    102.55 

I would like to detect whether the new value of disk space is an outlier or not. There are several outlier analysis methods (ref link):

  • Box plot analysis
  • Based on Z score
  • IQR based analysis (this is similar to box plot analysis)

Above methods are more statistical approach to detect outlier. There are several ways using an unsupervised machine learning algorithm to detect outlier (ref link). For example,

  • K-mean
  • Markov Chain
  • Isolation Forest etc.

Which method is suitable for the above dataset? Should I implement unsupervised based machine learning algorithm or statistical approach?

$\endgroup$
1
$\begingroup$

Given the specific context of detecting abnormal changes in the amount of free space, I'd suggest that you use the variation over time instead of the raw amount. For instance:

date                   free_space  variation
2019-05-15 09:00:00    102.65      NA
2019-05-15 09:05:00    102.69      0.04
2019-05-15 09:10:00    103.11      0.42
2019-05-15 09:15:00    102.58      -0.53
2019-05-15 09:20:00    102.55      -0.03

Whatever method you use, the variation is a much more relevant information to detect an unusual change than the raw size. You could also use a time window, e.g. calculate the variation over the last 30 minutes.

Personally I would simply use a heuristic for something like this: if the absolute value of the variation is higher than a threshold then label as outlier. The threshold could be a percentage of the size of the disk, e.g. 5%.

$\endgroup$
  • $\begingroup$ Thanks for your answer. I have the same idea to capture hour wise variation in data and apply box plot analysis or IQR over data. Can you please guide me: How can I use the unsupervised machine learning algorithm? Does it make sense? $\endgroup$ – Saurabh Chauhan May 16 at 12:46
  • $\begingroup$ Imho there's no point using sophisticated ML algorithms when your data consists in a single variable. boxplot/IQR is fine: it boils down to detecting when the value is above a threshold calculated on the whole dataset, which is similar to what I suggested. $\endgroup$ – Erwan May 16 at 13:07
  • $\begingroup$ Thanks for your suggestions. $\endgroup$ – Saurabh Chauhan May 16 at 13:12
1
$\begingroup$

If you are using R, you can screen for outliers by calculating Cook's distance.

Consider the following example. A hypothetical telecommunications company has information on the amount of data consumed by its customers (in mb), and wants to detect customers that have a much higher rate of usage than the general population.

# Compute Cooks Distance
dist <- cooks.distance(lm(usage ~ x1 + x2 + x3, data = trainset))
dist<-data.frame(dist)
s <- stdres(lm(usage ~ x1 + x2 + x3, data = trainset))
a <- cbind(trainset, dist, s)

The distance (dist) is displayed for each user along with their usage. In this case, the Cook's distance of 0.42 is significantly higher than for other users - indicating that this user is an outlier.

video

$\endgroup$
  • $\begingroup$ Thanks for your reply. I don't need any manual threshold. How can I do it automatically ? $\endgroup$ – Saurabh Chauhan May 17 at 6:53
  • $\begingroup$ You could calculate a percentage threshold for the Cook's distance. e.g. any distance with a threshold greater than 30% is automatically classified as an outlier, etc. $\endgroup$ – Michael Grogan May 17 at 10:55
  • $\begingroup$ Thanks for the quick response. Is it a thumb rule? How 30% threshold comes to picture? I am trying to understand $\endgroup$ – Saurabh Chauhan May 17 at 11:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.