As far as I know, when we deal with 'big data'. It is common we deal with more than 10 years' customers data. Quality issue is always there.
What I am thinking about, to what extend we should deal with the data quality issues? A robust algorithm should tolerate with the data quality issues but if we do better data cleansing work we get outcome confidently.
So how to balance this?
Somebody please give some ideas!
Thanks.
This is a question that is very dependent upon the data in question. Assuming that you can train your models in a reasonable time, I would start by not cleaning the data at all, and seeing how well your model performs, and then cleaning it a bit and redoing the experiment and so on. This is because it is possible to overclean your data, essentially removing variations in the data which are actually useful to your modelling. The amount of cleaning that you apply is yet another meta-variable in your model.
This all depends on the needs and the budget for the models. The first cleaning steps usually bring a decent increase in performance. The more steps you take the slower the improvements will increase in general. If you are doing something for yourself, cut it off at a certain point, if you are doing it for somebody else, ask them what they want and how much they are willing to pay for it. It's comparable to the 80/20 rule where the first 20% of the total work will help with 80% of the performance.
On another note, your statement about robust algorithms should tolerate with data qualities, you have to be careful with this. If there are biases in your data (missing values are not random for example) they will not learn properly, no matter how robust your algorithm is. Spending time using domain knowledge to fix this properly will help a lot.
I would say do as much cleaning and possible. The phrase "Garbage in, garbage out" is here for a reason.
Missing values, different definitions, suspicious information...all must be cleaned before fetched into any model.
Sure, even without all of this, the model will produce something, but it most likely will be less than optimal.
Data cleaning can be a real pain and also can easily take you away from the core task. Nevertheless, it is one of the critical aspect and hence cannot be taken lightly.
I believe you have the right idea because you mentioned that you are seeking balance. I always think of data cleaning as a marginal transaction. Beyond a certain point, it is not worth the time.
As Jan van mentioned, it all depends on your needs. If your model could be very sensitive to certain features, its better to have them cleaned religiously. If what you are looking for are broader insights, even a general clean of important features can work. The trick is to really understand the tipping point which can come with experience and / or knowledge of the data set.
There is this approach that my team takes often and has worked so far.
1. Once you know what model fits your problem, throw in a feature with random values.
2. Model your data set with minimal / basic cleaning.
3. Plot or tabulate accuracy for all features.
4. Drop all features that perform equal or worse than the random feature.
5. Some times depending on the accuracy even drop features that perform slightly higher than random.
Using this, we are mostly able to get rid of unnecessary features that hog up tidying effort.
Wash Rinse Repeat.
