0
$\begingroup$

I am doing univariate outlier detection in python. When I detect outliers for a variable, I know that the value should be whatever the highest non-outlier value is (i.e., the max if there were no outliers).

How can I impute this value in python or sklearn? I guess I can remove the values, get the max, replace the outliers and bring them back. But hoping there’s a function for that already.

Second, is this a bad idea? I see others remove the outlier completely or replace with the mean or median. I wonder if my approach is wrong.

$\endgroup$
0

2 Answers 2

2
$\begingroup$

My answer to the first question is use numpy's percentile function. And then, with y being the target vector and Tr the percentile level chose, try something like

import numpy as np
value = np.percentile(y, Tr)
for i in range(len(y)):
   if y[i] > value:
       y[i]= value

For the second question, I guess I would remove them or replace them with the mean if the outliers are an obvious mistake. But your approach seems reasonable otherwise.

$\endgroup$
1
  • $\begingroup$ Clean solution, thanks $\endgroup$
    – Insu Q
    Sep 25, 2019 at 14:03
2
$\begingroup$

Are you values stored in a dataframe? You could use .loc to do this also.

df.column_name.loc[df.column_name > max_value] = max_value

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.