How can I replace outliers with maximum non-outlier value?

Question

I am doing univariate outlier detection in python. When I detect outliers for a variable, I know that the value should be whatever the highest non-outlier value is (i.e., the max if there were no outliers).

How can I impute this value in python or sklearn? I guess I can remove the values, get the max, replace the outliers and bring them back. But hoping there’s a function for that already.

Second, is this a bad idea? I see others remove the outlier completely or replace with the mean or median. I wonder if my approach is wrong.

Answer 1

My answer to the first question is use numpy's percentile function. And then, with y being the target vector and Tr the percentile level chose, try something like

import numpy as np
value = np.percentile(y, Tr)
for i in range(len(y)):
   if y[i] > value:
       y[i]= value

For the second question, I guess I would remove them or replace them with the mean if the outliers are an obvious mistake. But your approach seems reasonable otherwise.

Answer 2

Are you values stored in a dataframe? You could use .loc to do this also.

df.column_name.loc[df.column_name > max_value] = max_value