Imagine that I'm a health department, and I have a data stream that describes whether someone gets sick after eating food from a restaurant. Each data point in the data stream arrives 24 hours after eating has occurred, and has the following data elements:

  1. Binary outcome: does the person get sick within 24 hours after eating at the restaurant?
  2. restaurant_type (categorical variable such as Chinese, Mexican, Italian, etc)
  3. restaurant_zip_code (categorical variable describing each location)
  4. order_type (categorical variable which could be of "dining in", "take out", "delivery by restaurant", "delivery by uber eats", "delivery by doordash")
  5. time_of_day (categorical variable describing time of day the order took place, which would be "6am-10am", "10am-2pm", "2pm-5pm", "5pm-10pm", "10pm-6am")

I want to create a model / algorithm that detects anomalies or unusual increases in sickness in the data stream. We expect that there would be some people randomly getting sick after eating food from a restaurant, so obviously we wouldn't want to investigate every instance of someone getting sick. But, let's say that there is a significant increase in sickness in people who "dined in" at Italian restaurants from a particular zip code from 2pm-5pm, then that would be unusual and I might want to investigate.

What is a good way to perform such a detection efficiently? One idea I thought of is I can generate many 1-D time series data based on each combination categorical values, and perform 1-D anomaly detection each. But, the number of combinations seems impractically large. For example, if there are 10 restaurant types, 40,000 zip codes, 5 order types, and 5 time_of_day ranges, that would be 10 * 40,000 * 5 * 5 = 10 million 1-D time series data that would need to be monitored.

Note: I'm not actually a health department. I am trying to efficiently perform anomaly detection on streaming data based on combinations of categories such as described by this similar "contrived" problem. Thank you!



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.