1
$\begingroup$

enter image description hereI have a univariate time series (there is a value for each time sampling) (sampling time: 66.66 micro second, number of samples/sampling time=151) coming from a scala customer

This time series contains some time frame which each of them are 8K (frequencies)*151 (time samples) in 0.5 sec [overall 1.2288 millions samples per half a second)

  • I need to find anomalous based on different rows (frequencies) Report the rows (frequencies) which are anomalous? (an unsupervised learning method)

Do you have an idea to which statistical parameter is more useful for it? or is it possible to evaluate without time sampling?

Do have an idea about suitable fast method (with lowest time delay: therefore some algorithm like kmeans does not work)

I should produce online automated ML

Improve this question
$\endgroup$

2 Answers 2

Reset to default
0
$\begingroup$

Is it possible to apply an offline anomaly pattern recognition? Or a "normal" pattern recognition that gives an alert when it is out of its boundaries?

If yes, it could be the best option as you only need to detect specific behavior with some error margin and without heavy calculations.

Low level processes with high sampling speed generally have simple dynamics that don't require complex algorithms.

Improve this answer
$\endgroup$
9
  • $\begingroup$ thanks but the system is online learning. As I said the input come from scala message bus and speed of finding the anomalies is important? Have you any idea about it? $\endgroup$
    – user10296606
    Jul 30, 2021 at 0:44
  • $\begingroup$ Is it possible to have an example (even an invented one) of the data? In all the case, you can update your model with every new sequence. Do you have to develop it in Scala? $\endgroup$
    – Nicolas Martin
    Jul 30, 2021 at 7:22
  • $\begingroup$ transaction with environment should be in scala (because of speed) I can send you an example of data (where I can send it?) Yes, it should be updated but I should decide about memory and others (see below) datascience.stackexchange.com/questions/98189/… $\endgroup$
    – user10296606
    Aug 6, 2021 at 3:56
  • $\begingroup$ You can modify your original question by adding a very small table sample like [[0.001, 0.032,...,0.0,0.26],[0.56,0.11,...,0.67,0.21],....,[....]] and you can slighty change numbers if there are confidentiality restrictions. $\endgroup$
    – Nicolas Martin
    Aug 6, 2021 at 7:31
  • $\begingroup$ It is too much to write them but overall it is for instance 110.563 something like it and it is 8K (8192) rows which each of them has 151 elements (each element is something like: 110.563) size: 8192*151=1.2288 M I add a photo of based on min of each 151 elements $\endgroup$
    – user10296606
    Aug 10, 2021 at 4:12
0
$\begingroup$

To start simple, we can keep a stat of the mean and std of each frequency band from past observations, and compute the likelihood of observing the new incoming frequencies (assuming some distribution).

If the anomaly pattern is more complicated, you may have to resort to other standard unsupervised learning techniques, K-means for example.

Any prior knowledge of anomaly pattern helps simplify the problem, so make sure you look for it.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.