Outlier detection for Disk Space Usage

Question

I would like to do outlier or anomaly detection on the disk free space data. Sample dataset as below (I don't have any label dataset):

date                   free_space (GB)
2019-05-15 09:00:00    102.65
2019-05-15 09:05:00    102.69
2019-05-15 09:10:00    103.11
2019-05-15 09:15:00    102.58
2019-05-15 09:20:00    102.55 

I would like to detect whether the new value of disk space is an outlier or not. There are several outlier analysis methods (ref link):

• Box plot analysis
• Based on Z score
• IQR based analysis (this is similar to box plot analysis)

Above methods are more statistical approach to detect outlier. There are several ways using an unsupervised machine learning algorithm to detect outlier (ref link). For example,

• K-mean
• Markov Chain
• Isolation Forest etc.

Which method is suitable for the above dataset? Should I implement unsupervised based machine learning algorithm or statistical approach?

Answer 1

Given the specific context of detecting abnormal changes in the amount of free space, I'd suggest that you use the variation over time instead of the raw amount. For instance:

date                   free_space  variation
2019-05-15 09:00:00    102.65      NA
2019-05-15 09:05:00    102.69      0.04
2019-05-15 09:10:00    103.11      0.42
2019-05-15 09:15:00    102.58      -0.53
2019-05-15 09:20:00    102.55      -0.03

Whatever method you use, the variation is a much more relevant information to detect an unusual change than the raw size. You could also use a time window, e.g. calculate the variation over the last 30 minutes.

Personally I would simply use a heuristic for something like this: if the absolute value of the variation is higher than a threshold then label as outlier. The threshold could be a percentage of the size of the disk, e.g. 5%.

Answer 2

If you are using R, you can screen for outliers by calculating Cook's distance.

Consider the following example. A hypothetical telecommunications company has information on the amount of data consumed by its customers (in mb), and wants to detect customers that have a much higher rate of usage than the general population.

# Compute Cooks Distance
dist <- cooks.distance(lm(usage ~ x1 + x2 + x3, data = trainset))
dist<-data.frame(dist)
s <- stdres(lm(usage ~ x1 + x2 + x3, data = trainset))
a <- cbind(trainset, dist, s)

The distance (dist) is displayed for each user along with their usage. In this case, the Cook's distance of 0.42 is significantly higher than for other users - indicating that this user is an outlier.