0
$\begingroup$

I have a dataset which has a column of prices, a column of dates, and various other columns of numerical and categorical values. I would like to find outlier prices based on all the columns in the dataset, and to this end I have settled on using an isolation forest. The question I have though is how do I find outlier prices using an isolation forest?

As an example, I have a dataframe iso_df with 5 columns: PRICE is a column of prices which I want to find outliers for, RIV_VALUES is a second column of prices (but I don't want to find outliers for this), ITEM_GROUP is a categorical column with values such as A, B, C, etc., and YEAR_MONTH is a column of year-month combinations (e.g. 2020-01, 2020-02, etc.) so the example dataframe looks like this:

  PRICE    |   RIV_VALUES   |  ITEM_GROUP  |    YEAR_MONTH
---------------------------------------------------------------
    2.2    |      4.7       |       A      |     2020-01
    7.9    |      3.9       |       A      |     2020-02  
     2     |      1.2       |       B      |     2020-01
    10.2   |      8.9       |       A      |     2020-03
    25.2   |      2.4       |       D      |     2020-02

i.e.

iso_df = iso_df[['PRICE', "RIV_VALUES", "ITEM_GROUP", 'YEAR_MONTH']]

After this, I use label_encoder from the scikit-learn package to transform the ITEM_GROUP and YEAR_MONTH columns to ordinal numerical values.

Once that is done, my idea was doing something like this:

from sklearn.ensemble import IsolationForest

clf = IsolationForest(n_estimators = 10, max_samples='auto', contamination=float(.05), max_features=1.0, bootstrap=False, n_jobs=-1, verbose=0)
clf_prediction = clf.fit_predict(iso_df)

However, this feels incorrect to me - since I'm trying to find PRICES which are outliers, should I still combine it all in the model like this? Or should I find a way to separate them more? Or alternatively, is there a better model to do this with (besides the usual confidence interval/box plot approaches). Any help would be appreciated, thanks in advance!

$\endgroup$

1 Answer 1

0
$\begingroup$

Be aware that you have a level feature, ÌTEM_GROPUP. It implies that what may be considered an outlier for category A may not be for category B or C. You could make some descriptive analysis of each category to check how similar the distribution for each category is.

You may have some alternatives:

  1. Treat the problem as a time series anomaly detection where you will have the historical prices over time, so you will only include PRICE in your model.

  2. If you have enough data for every category, you could train an independent anomaly detection model for each category. So in the inference stage, you will first check the new point's category and then apply the corresponding model.

I hope it helps!

$\endgroup$
1
  • $\begingroup$ Thanks for your reply! Regarding your comment about the ITEM_GROUP column, wouldn't the Isolation Forest be able to do that for me? I was under the impression that since Isolation Forests can work with categorical data, they can determine outliers based on each individual category (i.e. A, B, etc.), or is my understanding wrong? If my understanding is indeed incorrect, then I will definitely try out your second alternative! $\endgroup$ Apr 12, 2023 at 0:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.