# Outlier detection for Disk Space Usage

I would like to do outlier or anomaly detection on the disk free space data. Sample dataset as below (I don't have any label dataset):

date                   free_space (GB)
2019-05-15 09:00:00    102.65
2019-05-15 09:05:00    102.69
2019-05-15 09:10:00    103.11
2019-05-15 09:15:00    102.58
2019-05-15 09:20:00    102.55


I would like to detect whether the new value of disk space is an outlier or not. There are several outlier analysis methods (ref link):

• Box plot analysis
• Based on Z score
• IQR based analysis (this is similar to box plot analysis)

Above methods are more statistical approach to detect outlier. There are several ways using an unsupervised machine learning algorithm to detect outlier (ref link). For example,

• K-mean
• Markov Chain
• Isolation Forest etc.

Which method is suitable for the above dataset? Should I implement unsupervised based machine learning algorithm or statistical approach?

Given the specific context of detecting abnormal changes in the amount of free space, I'd suggest that you use the variation over time instead of the raw amount. For instance:

date                   free_space  variation
2019-05-15 09:00:00    102.65      NA
2019-05-15 09:05:00    102.69      0.04
2019-05-15 09:10:00    103.11      0.42
2019-05-15 09:15:00    102.58      -0.53
2019-05-15 09:20:00    102.55      -0.03


Whatever method you use, the variation is a much more relevant information to detect an unusual change than the raw size. You could also use a time window, e.g. calculate the variation over the last 30 minutes.

Personally I would simply use a heuristic for something like this: if the absolute value of the variation is higher than a threshold then label as outlier. The threshold could be a percentage of the size of the disk, e.g. 5%.

• Thanks for your answer. I have the same idea to capture hour wise variation in data and apply box plot analysis or IQR over data. Can you please guide me: How can I use the unsupervised machine learning algorithm? Does it make sense? May 16 '19 at 12:46
• Imho there's no point using sophisticated ML algorithms when your data consists in a single variable. boxplot/IQR is fine: it boils down to detecting when the value is above a threshold calculated on the whole dataset, which is similar to what I suggested. May 16 '19 at 13:07
• Thanks for your suggestions. May 16 '19 at 13:12

If you are using R, you can screen for outliers by calculating Cook's distance.

Consider the following example. A hypothetical telecommunications company has information on the amount of data consumed by its customers (in mb), and wants to detect customers that have a much higher rate of usage than the general population.

# Compute Cooks Distance
dist <- cooks.distance(lm(usage ~ x1 + x2 + x3, data = trainset))
dist<-data.frame(dist)
s <- stdres(lm(usage ~ x1 + x2 + x3, data = trainset))
a <- cbind(trainset, dist, s)


The distance (dist) is displayed for each user along with their usage. In this case, the Cook's distance of 0.42 is significantly higher than for other users - indicating that this user is an outlier. • Thanks for your reply. I don't need any manual threshold. How can I do it automatically ? May 17 '19 at 6:53
• You could calculate a percentage threshold for the Cook's distance. e.g. any distance with a threshold greater than 30% is automatically classified as an outlier, etc. May 17 '19 at 10:55
• Thanks for the quick response. Is it a thumb rule? How 30% threshold comes to picture? I am trying to understand May 17 '19 at 11:48