1
$\begingroup$

I should decide on the contamination value while using the Isolation Forests algorithm (I am using the sklearn implementation). Otherwise, sklearn's default is 0.1.

I am worried if I decide for this parameter with experimenting on my current dataset, it will differ by time as I would be receiving new data points very soon.

How can I know that this parameter better than any other, just by experimenting it with outlier removal?

Thanks.

$\endgroup$
1
$\begingroup$

If you want to experiment something like that I think you can add some outliers into your data for that respective field and see what are its effects.

If you find that these are prone to outliers(most of of the features are), then you can handle situation by removing such outliers before appending to the existing data.

What is your target variable? Did you do any kind of Correlation analysis/ Predictor Importance?

| improve this answer | |
$\endgroup$
  • $\begingroup$ Adding outliers is rather complicated in my case since I have 130 dimensions and my aim is to detect the outliers that I could not detect with just a general analysis on the obvious attributes. I have a regression problem and my target variable is continues values with a wide range (I use gradient boosting regression trees). I applied some correlation analysis for feature reduction but I am not sure if this is what you meant. $\endgroup$ – mari Nov 15 '17 at 12:32
  • $\begingroup$ The reason why I suggested that was to see if they are prone to outliers or not. Do you know the target variable? $\endgroup$ – Toros91 Nov 15 '17 at 13:07
  • $\begingroup$ Yes, I know the target variable but don't know nor visualize the outliers. $\endgroup$ – mari Nov 16 '17 at 10:06
  • $\begingroup$ Apply Boruta package, give the target variable and it will give you predictor importance and it will tell you the important features. $\endgroup$ – Toros91 Nov 16 '17 at 10:13
  • $\begingroup$ @mari use cluster analysis to find out the outliers $\endgroup$ – Toros91 Nov 16 '17 at 10:16
1
$\begingroup$

Okay, so probably this was obvious but here is what I did. I used the

scores = decision_function(x)

instead of just predicting the anomalies, and obtained the anomaly scores for each instance. Afterwards, I could draw the anomaly distribution of the dataset and set a better outlier proportion regarding the distribution.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Would you mind elaborating on this answer a bit more? What does the decision_function give you and how do you translate it to a firm value like 0.1? $\endgroup$ – Zhubarb Jan 17 '19 at 9:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.