Isolation forest sklearn contamination param

Question

I am working on an unsupervised anomaly detection task on time series data using an isolation forest algorithm. I am developing it in Python, more in detail using scikit-learn.

I found a lot of examples on this, but what is not very clear, is how to set the contamination parameter during the instantiation of IsolationForest.

Looking at the documentation, contamination is

the amount of contamination of the data set, i.e. the proportion of outliers in the data set.

Should I use some statistical techniques to identify this percentage?

Answer 1

I think the answer is "it depends". The contamination parameter simply controls the threshold for the decision function when a scored data point should be considered an outlier. It has no impact on the model itself.

Depending on your application it could make sense to use some statistical analysis to get a rough estimate of the contamination. (If you could narrow it down more specifically then you wouldn't need the isolation forest...)

Usually, I would say, this is informed by some business expectation. Something like "we expect to see a similar rate to our competitors who have reported a rate of x".

Or you expect a certain number of outliers in your dataset. Then you can use the raw scores to find a threshold that gives you that number and set the contamination parameter retrospectively when applying the model to new data.

Answer 2

The parameter contamination is the (suspected) "share" of outliers in the dataset. Consider the example:

from sklearn.ensemble import IsolationForest
x = [[-1], [2], [3], [5], [7], [10], [12], [20], [30], [100]]

An IsolationForest with contamination=0.1 would identify the "top 10" percent of outliers.

clf = IsolationForest(contamination=0.1).fit(x)
clf.predict(x)
> [ 1  1  1  1  1  1  1  1  1 -1]

The -1 indicates an "outlier" here.

Setting contamination=0.2 yields "two out of ten" values identified as outliers.

clf = IsolationForest(contamination=0.2).fit(x)
clf.predict(x)
> [1  1  1  1  1  1  1  1 -1 -1]

The parameter contamination usually needs to be tuned (or otherwise assumptions need to be made). When tuned, it would usually be set to a value that optimizes some objective of a top level estimator (after removing outliers using IsolationForest).

Answer 3

I am masking my data with the result from the prediction then, impute the false recovery with zero; by counting the zeros I could have an estimation of the contamination.