How to detect Anomalies in raw level data?

Question

I have a dataset that is growing at about 30 million rows per day.

The data schema is follow:

day_hour_timestamp timestamp,
campaign_id        varchar,
creative_id        int,
uuid               varchar,
device             varchar,
country            varchar,
os                 varchar,
browser            varchar

Each record in the dataset is information about an advertisement impression.

For example, if I visited a particular advertisement at a website on 10/11/2017 13:30. I will be one datapoint in the dataset for 10/11/2017 hour 13:00 along with my device, country, os, browser, etc.

My question is how do detect anomalies within this dataset? Anomalies such as data not normal to the columns, for example if the I receive an OS that is NULL or browser is 'Apple'.

Should I create a heuristic that checks every column or is there a machine learning algorithm I can apply?

Since this dataset is pretty big would checking each data point be expensive?

Answer 1

Should I create a heuristic that checks every column or is there a machine learning algorithm I can apply?

I would go with unsupervised or semi-supervised learning. Use 'normal' data as training data and then predict what is normal and what is abnormal. Depending on your needs you can fine-tune the cut-off.

One-class SVM should work. Or a combination of one-class SVM and SV Data-Description for possibly better results.

Answer 2

How I go about anomaly detection is having a domain knowledge. In your case there are multiple domains as columns so you can use Ontologies as a way to detect anomalies. There are multiple resources for that like WordNet, ConceptNet and GloVe which provide extensive data. You can get what classes the query belongs to and check if the class you want is present.