# Anomaly Detection and Removal/Interpolate [closed]

I am performing a machine learning regression task on time series data. I have a data frame filled with the close prices of various assets and economic data. I am looking to perform outlier detection on the entire dataframe. I have three questions:

• What are some good methods to perform this?
• Any packages that are good for this?
• With the outliers, I don't want to drop a whole row as this will lose data points for the other time series in the dataset, so can I fill those anomalous datapoints with NaN values and then interpolate between the data points?

Here is a subsection of my dataframe (df):

              audjpy    audnzd    audusd   usdcad   cadchf      cadjpy
2008-01-01  98.050003  1.140000  0.877116  0.99320  1.13960  112.599998
2008-01-02  96.559998  1.139500  0.884017  0.99440  1.12560  109.820000
2008-01-03  96.550003  1.140900  0.881601  0.98880  1.12370  110.693001
2008-01-04  95.168999  1.138300  0.876578  0.99880  1.10470  108.459999
2008-01-07  95.220001  1.134500  0.871916  1.00550  1.11130  108.559998
...         ...         ...      ...       ...     ...       ....


1. If your signal is normally distributed (or very close to normal distribution) you can remove points (or replace by NaN) all those points lying outside $$[\mu - n\sigma, \mu + n\sigma]$$ where $$\mu$$ is the mean of the signal, $$\sigma$$ its standard deviation and $$n$$ is to be examined (you can try plotting the points histogram and examine closely the distribution).
2. A more sophisticated approach is to pass a rolling mean filter on the signal to capture the trend, and compute the residual signal = original - trend. And then clean the residual (If you plot the residual distribution, you will see it is most probably unimodal -- sometimes symmetric ... , so cleaning such a signal can be nicely done.) You compute the IQR (interquartile range) of your residual and remove points lying outside $$[q_1 - n*IQR, q_3 + n * IQR]$$ where $$q_1$$ and $$q_3$$ denote the first and third quantiles of the residual signal (respectively), $$n$$ typically people use 2 or 3...