1
$\begingroup$

I should decide on the contamination value while using the Isolation Forests algorithm (I am using the sklearn implementation). Otherwise, sklearn's default is 0.1.

I am worried if I decide for this parameter with experimenting on my current dataset, it will differ by time as I would be receiving new data points very soon.

How can I know that this parameter better than any other, just by experimenting it with outlier removal?

Thanks.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

If you want to experiment something like that I think you can add some outliers into your data for that respective field and see what are its effects.

If you find that these are prone to outliers(most of of the features are), then you can handle situation by removing such outliers before appending to the existing data.

What is your target variable? Did you do any kind of Correlation analysis/ Predictor Importance?

Improve this answer
$\endgroup$
7
  • $\begingroup$ Adding outliers is rather complicated in my case since I have 130 dimensions and my aim is to detect the outliers that I could not detect with just a general analysis on the obvious attributes. I have a regression problem and my target variable is continues values with a wide range (I use gradient boosting regression trees). I applied some correlation analysis for feature reduction but I am not sure if this is what you meant. $\endgroup$
    – mari
    Nov 15, 2017 at 12:32
  • $\begingroup$ The reason why I suggested that was to see if they are prone to outliers or not. Do you know the target variable? $\endgroup$
    – Toros91
    Nov 15, 2017 at 13:07
  • $\begingroup$ Yes, I know the target variable but don't know nor visualize the outliers. $\endgroup$
    – mari
    Nov 16, 2017 at 10:06
  • $\begingroup$ Apply Boruta package, give the target variable and it will give you predictor importance and it will tell you the important features. $\endgroup$
    – Toros91
    Nov 16, 2017 at 10:13
  • $\begingroup$ @mari use cluster analysis to find out the outliers $\endgroup$
    – Toros91
    Nov 16, 2017 at 10:16
1
$\begingroup$

Okay, so probably this was obvious but here is what I did. I used the

scores = decision_function(x)

instead of just predicting the anomalies, and obtained the anomaly scores for each instance. Afterwards, I could draw the anomaly distribution of the dataset and set a better outlier proportion regarding the distribution.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Would you mind elaborating on this answer a bit more? What does the decision_function give you and how do you translate it to a firm value like 0.1? $\endgroup$
    – Zhubarb
    Jan 17, 2019 at 9:27
  • $\begingroup$ @mari Thanks for your great answer. May I ask you kindly to have a look at related post here? $\endgroup$
    – Mario
    Mar 16, 2021 at 12:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.