Anomaly Detection and Removal/Interpolate [closed]

Question

I am performing a machine learning regression task on time series data. I have a data frame filled with the close prices of various assets and economic data. I am looking to perform outlier detection on the entire dataframe. I have three questions:

• What are some good methods to perform this?
• Any packages that are good for this?
• With the outliers, I don't want to drop a whole row as this will lose data points for the other time series in the dataset, so can I fill those anomalous datapoints with NaN values and then interpolate between the data points?

Here is a subsection of my dataframe (df):

              audjpy    audnzd    audusd   usdcad   cadchf      cadjpy  
2008-01-01  98.050003  1.140000  0.877116  0.99320  1.13960  112.599998   
2008-01-02  96.559998  1.139500  0.884017  0.99440  1.12560  109.820000   
2008-01-03  96.550003  1.140900  0.881601  0.98880  1.12370  110.693001   
2008-01-04  95.168999  1.138300  0.876578  0.99880  1.10470  108.459999   
2008-01-07  95.220001  1.134500  0.871916  1.00550  1.11130  108.559998
...         ...         ...      ...       ...     ...       ....

Answer 1

In my opinion, I would treat each signal on its own. The approach also depends on the signals and on your definition of anomalies/outliers (for example unexpected long peaks?). But I can point some methods that you can try if they work on your signals:

1. If your signal is normally distributed (or very close to normal distribution) you can remove points (or replace by NaN) all those points lying outside $[\mu - n\sigma, \mu + n\sigma]$ where $\mu$ is the mean of the signal, $\sigma$ its standard deviation and $n$ is to be examined (you can try plotting the points histogram and examine closely the distribution).
2. A more sophisticated approach is to pass a rolling mean filter on the signal to capture the trend, and compute the residual signal = original - trend. And then clean the residual (If you plot the residual distribution, you will see it is most probably unimodal -- sometimes symmetric ... , so cleaning such a signal can be nicely done.) You compute the IQR (interquartile range) of your residual and remove points lying outside $[q_1 - n*IQR, q_3 + n * IQR]$ where $q_1$ and $q_3$ denote the first and third quantiles of the residual signal (respectively), $n$ typically people use 2 or 3...

Of course when you detect an outlier, you can replace the value by NaN and interpolate later. That should be totally fine ...

Finally, many other methods exist and a lot has been done, I hope that my answer will help you and even motivate other approaches.