How do you assess the quality of your data? In data scientists' world, we come across several data. We often crunch numbers without formally assessing its quality due to various reasons. One such reason is we need to meet deadlines for reports & publications. I am wondering if anyone has adopted or come across a method/guidelines that help to find issues within the data (time-saving tips), so we can analyze the data efficiently. Please share your experience, tips, etc.
This entirely depends on what you wish or aim to do with your data, and what you mean by your tag
data-cleaning. This can involve the technical process of sanitising data i.e. fixing broken
XML, but also the manual deletion of outliers etc.
Are you going to perform statistics to measure the correlation between certain variables?
Or: Are you going to include the data into a machine-learning solution?
In any case, you can plot descriptive statistics to get an impression of the quality of your data, such as:
- Box-plots to detect outliers
- Standard deviation, mean, mode, median to assess if it's normally distributed or not
- More, don't know from the top of my head
Remember that those measures give you an impression of the quality of your data, but even good measurements can come from bad data, and possibly vice versa.
In short, the best way of assessing the quality of your data is to assess the quality of your results, which are acquired generally after several experiments. Only then will you see how well your data is suited for the problem you are trying to solve. That - or you need to get your hands dirty and examine the data qualitatively before you use it.
Data (in assumingly large quantities) are generally used to identify certain patterns within the phenomena described by the data. It is very important to describe your task in terms of a (research) question and a proposed solution or hypothesis. Then, you can assess whether the data is suitable for your particular (research) project. Keep in mind that, theoretically, you can try to answer numerous questions on the same data, or answer the same question on different data sets. The goal is the right balance.
Your question is very broad, and therefore I won't offer very specific answers. You asked for "time-saving tips" but there are many and they depend on context.
Instead, I'll offer a general set of heuristics that are useful most of the time.
- Start with a specification or definition of what "quality" means for your analysis, and therefore for your data.
- Use definition/specification in 1) to enumerate the types of errors, omissions, missteps, modifications, etc. that might arise in the course of data collection and recording. This enumeration will always be provisional because there could be ways that quality could be degraded that you didn't think of originally.
- Use 2) to define inspection and test methods that might reveal the existence of any of these types of errors, omissions, etc. The default method is "eyeballing" -- having a knowledgeable, experienced person looking at the data to see if it looks right.
- When you uncover errors, omissions, etc. try to determine the root causes and generating processes. E.g. erroneous data in dozens of columns resulted from an "off-by-one" bug in the ETL job that generated/translated the data from source files.
- Using 2), 3) and 4), define processes that might either correct erroneous data or mitigate the effects of data quality problems. E.g. Soundex transformation of names can mitigate effects of misspelling, but can't mitigate people misusing the "name" field to enter notes: e.g. "Nelson - DO NOT CALL". Be aware that any process you might use to correct or mitigate errors or problems might create new errors or problems (e.g. truncation of numerical data).
- After you perform your analysis, look back at the data and ask "Could these results be the result of data quality problems rather than true/accurate/appropriate data?" In other words, double check.