0
$\begingroup$

I'm a little bit stuck on how to efficiently model anomaly detection for the following problem, probably because of my lack of experience with time series modelling:

  • I retrieve market data sorted by timestamp containing the securities ID and a corresponding timestamp/price pair for each transaction. This also means that the preprocessing step of converting the data into time series (corresponding each to a security) is going to be costly, since there are usually a couple million transactions per day.
  • This is done over a very large set of securities, usually spanning a single day. The amount of transactions of a single security can vary wildly, from being traded once per day/week/month up to multiple times per second.
  • The goal is to find outliers/anomalies concerning the price development of individual securities, i.e. a blunderbuss attempt with a clustering method might not detect outliers lying inside clusters but "outside" their respective time series.
  • I plan to turn this into real-time/online code, so it would be very convenient if the methodology from the offline case (done using archived data from a few days ago) carries over. RAM should not be a concern.
  • Bonus points if the approach does not explicitly make use of time series, i.e. I can use the data I have at hand with no/not much further costly preprocessing (I know this sound unlikely). Extra bonus points if there is an implementation available using Python, but this is only for convenience of testing.

I have already tried to sketch up methods using AR(I)MA, agglomerative clustering along timesteps, DWT and LOF, but they always fail in at least one step of their conception (high cost, no canonical way of combining clusters along timesteps, too much preprocessing).

What is an approach maximizing the efficiency-accuracy tradeoff for this problem? Is there a corresponding name for the algorithm describing that approach?

$\endgroup$

1 Answer 1

0
$\begingroup$

Sounds like a super interesting problem. One detection algorithm I used recently to some success when attempting to find "non-artifact" anomalies in EEG data was the "spectral residual" from the Time-Series Anomaly Detection Service at Microsoft.

Essentially, it does some clever manipulation of the Fourier Transformed data in order to highlight the innovations/anomalies. The maths is quite simple and I managed to implement it in a few lines of numpy. I can post the code if you are interested.

To fit this in an online context, I calculated saliency maps on 200 seconds of 128 Hz samples (akin to model training). I fed these data-points as priors to a Bayesian change-point detection algorithm, which I then used to calculate the probability if data from the next second of saliency maps was a "change-point."

Of course, this is last part is a bit of a departure from what you're aiming at, but I think even a simple moving average comparison would do the trick.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.