I have a dataset that is growing at about 30 million rows per day.
The data schema is follow:
Each record in the dataset is information about an advertisement impression.
For example, if I visited a particular advertisement at a website on 10/11/2017 13:30. I will be one datapoint in the dataset for 10/11/2017 hour 13:00 along with my device, country, os, browser, etc.
My question is how do detect anomalies within this dataset? Anomalies such as data not normal to the columns, for example if the I receive an OS that is NULL or browser is 'Apple'.
Should I create a heuristic that checks every column or is there a machine learning algorithm I can apply?
Since this dataset is pretty big would checking each data point be expensive?