Could someone please suggest how to remove local outliers from the dataframe? I have the code to detect the local outliers, but I need help removing them(setting these values to zero) in the dataframe. Any advice would be highly appreciated.

The code to detect the local outliers is below:

def printOutliers(series, window, scale= 1.96, print_outliers=False):

rolling_mean = series.rolling(window=window).mean()

#Print indices of outliers
if print_outliers:
    mae = mean_absolute_error(series[window:], rolling_mean[window:])#mean absolute error is a measure of difference between two continuous variables. 
    deviation = 3*np.std(series[window:] - rolling_mean[window:])
    lower_bound = rolling_mean - (mae + scale * deviation)
    upper_bound = rolling_mean + (mae + scale * deviation)
    outliers_lower = series[series<lower_bound]
    outliers_upper = series[series>upper_bound]
    print("values beyond lower bound are: " +  "\n"  + str(outliers_lower))
    print("values beyond lower bound are: " + "\n" + str(outliers_upper))  

printOutliers(df['Column1'].dropna(how='any'), 10, print_outliers=True)

You could use this approach here:

# accept a dataframe, remove outliers, return cleaned data in a new dataframe
# see http://www.itl.nist.gov/div898/handbook/prc/section1/prc16.htm
def remove_outlier(df_in, col_name):
    q1 = df_in[col_name].quantile(0.25)
    q3 = df_in[col_name].quantile(0.75)
    iqr = q3-q1 #Interquartile range
    fence_low  = q1-1.5*iqr
    fence_high = q3+1.5*iqr
    df_out = df_in.loc[(df_in[col_name] > fence_low) & (df_in[col_name] < fence_high)]
    return df_out
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.