Multivariate outlier detection with isolation forest..How to detect most effective features?

Question

I am trying to detect outliers in my data-set with 5000 observations and 800 features. I have followed the simple steps told in http://scikit-learn.org/stable/auto_examples/ensemble/plot_isolation_forest.html

There are some examples in stackoverflow and other sources as well, however, I could not find a decent explanation about the interpretation of outliers returned by isolationforest. First, what I did is:

from sklearn.ensemble import IsolationForest

X_train = trbb[check_cols]

clf = IsolationForest(n_jobs=6,n_estimators=500, max_samples=256, random_state=23)
clf.fit(X_train)
y_pred_train = clf.predict(X_train)
y_pred_train

This returns array([1, 1, 1, ..., 1, 1, 1]) where -1's are the outliers.

The shape of y_pred_train is 5000, which is identical with X_train[0]. So the index of -1 corresponds to the index of X_train. For exp. one of the outlier indices returned by IF is 532. So this means that the point (in 800 dimensional space) in this index is detected as an outlier. After detecting this, how can I approach and use the results to dig more? For example, can I reach the most important features causing the outliers?

To this end, I applied PCA and used the newly created (50) components as new variables. Then, by plotting component pairs with -1 & 1s returned by IF, I tried to get some insight of possible outliers. Indeed, I observed some points as shown below:

Here, the black dots are -1s (outliers) and the yellow ones are 1s (inliers). I interpret this that if both components have very high values, it might cause an outlier. However, how can I decide which original values are the most effective ones for those outliers? For example, can I use the original variables with the highest weights (let's pick top 10) in the related components? Or could you suggest any other method?

Thanks in advance!

Answer 1

One important difference between isolation forest and other types of decision trees is that it selects features at random and splits the data at random, so it won't produce a nice feature importance list; and the outliers are those that end up isolated with fewer splits or who end up in terminal nodes with few observations.

I guess if you dig into the code you might be able to rank the features according to levels of the splits that use them and the number of observations in the nodes, but AFAIK neither the scikit-learn implementation, nor Zelazny's R implementation, have such a thing built-in (although the R one has some functions for individual nodes: https://github.com/Zelazny7/isofor/blob/master/R/interpret.R).

Nevertheless, after you identify the outliers from your sample, you can use that as labeled data for a classification problem, and from there fit a model such as xgboost or random forest that could give you the feature importances.