10
$\begingroup$

I am working on an unsupervised anomaly detection task on time series data using an isolation forest algorithm. I am developing it in Python, more in detail using scikit-learn.

I found a lot of examples on this, but what is not very clear, is how to set the contamination parameter during the instantiation of IsolationForest.

Looking at the documentation, contamination is

the amount of contamination of the data set, i.e. the proportion of outliers in the data set.

Should I use some statistical techniques to identify this percentage?

$\endgroup$

3 Answers 3

3
$\begingroup$

I think the answer is "it depends". The contamination parameter simply controls the threshold for the decision function when a scored data point should be considered an outlier. It has no impact on the model itself.

Depending on your application it could make sense to use some statistical analysis to get a rough estimate of the contamination. (If you could narrow it down more specifically then you wouldn't need the isolation forest...)

Usually, I would say, this is informed by some business expectation. Something like "we expect to see a similar rate to our competitors who have reported a rate of x".

Or you expect a certain number of outliers in your dataset. Then you can use the raw scores to find a threshold that gives you that number and set the contamination parameter retrospectively when applying the model to new data.

$\endgroup$
2
  • $\begingroup$ Thank you for the answer! can you please give me some examples of statistical analysis that i can apply? thanks $\endgroup$
    – Giordano
    Jul 4, 2019 at 21:42
  • 1
    $\begingroup$ I'd probably need to know more about your data. Maybe ask a new question? :) $\endgroup$
    – oW_
    Jul 8, 2019 at 20:06
2
$\begingroup$

The parameter contamination is the (suspected) "share" of outliers in the dataset. Consider the example:

from sklearn.ensemble import IsolationForest
x = [[-1], [2], [3], [5], [7], [10], [12], [20], [30], [100]]

An IsolationForest with contamination=0.1 would identify the "top 10" percent of outliers.

clf = IsolationForest(contamination=0.1).fit(x)
clf.predict(x)
> [ 1  1  1  1  1  1  1  1  1 -1]

The -1 indicates an "outlier" here.

Setting contamination=0.2 yields "two out of ten" values identified as outliers.

clf = IsolationForest(contamination=0.2).fit(x)
clf.predict(x)
> [1  1  1  1  1  1  1  1 -1 -1]

The parameter contamination usually needs to be tuned (or otherwise assumptions need to be made). When tuned, it would usually be set to a value that optimizes some objective of a top level estimator (after removing outliers using IsolationForest).

$\endgroup$
1
  • $\begingroup$ how to tune though? when you do not know anything about scoring function in an unsupervised fashion? $\endgroup$
    – ARAT
    yesterday
0
$\begingroup$

I am masking my data with the result from the prediction then, impute the false recovery with zero; by counting the zeros I could have an estimation of the contamination.

$\endgroup$
1
  • $\begingroup$ As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Nov 21, 2021 at 1:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.