3
$\begingroup$

I have a time series data which is available in offline csv format. I am using this data to create anomaly detection model. Although I could create this model to predict anomalies in this dataset, I need to use this model when data is real time. Every second data point will be coming and then if point is anomalous I need model to raise an anomaly.

Any ideas how should I implement this?

Edit - Data is updated regularly in computer, file is updated as data is streaming in this file only. Predictions are required for the realtime anomaly detection as decision will have to be made to stop the process when such prediction for detection of anomaly is made.

$\endgroup$
1
$\begingroup$

It depends on the answer to these questions:

  • How is the real-time data provided?
  • How often and in which format are the predictions required?

It's common to use a database in such a setting. So the ML prediction process would be called regularly, query this database to get the latest instances and generate predictions in the required format (possibly storing the predictions in the database as well).

| improve this answer | |
$\endgroup$
  • $\begingroup$ 1. Data is updated regularly in computer, file is updated as data is streaming in this file only. 2. Predictions are required for the realtime anomaly detection as decision will have to be made to stop the process when such prediction for detection of anomaly is made. $\endgroup$ – Bhakti Jun 13 '19 at 7:53
  • $\begingroup$ So your process could consist in doing the following every N seconds: (1) extract instances from the file as a csv; (2) run the model and write the result to an output file; (3) the main process regularly checks the output file and stops if an anomaly was detected. Of course you need to make sure that the process takes less than N seconds. On linux systems you could use cron to repeat the process every N seconds. $\endgroup$ – Erwan Jun 13 '19 at 10:43
0
$\begingroup$

I had similar situation, to mimic file data as if they are streaming in real time. You can use ‘schedule’/time library to read file every x seconds/milliseconds. You need to put the entire operation/prediction section of the code in a function which ‘schedule’ will be triggering every x secs. Hope this helps.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.