# How do you ascertain the quality of your data? [closed]

How do you assess the quality of your data? In data scientists' world, we come across several data. We often crunch numbers without formally assessing its quality due to various reasons. One such reason is we need to meet deadlines for reports & publications. I am wondering if anyone has adopted or come across a method/guidelines that help to find issues within the data (time-saving tips), so we can analyze the data efficiently. Please share your experience, tips, etc.

• This question is too broad and needs to be framed as a question rather than a seed for a discussion. – Nitesh May 26 '15 at 23:34

This entirely depends on what you wish or aim to do with your data, and what you mean by your tag data-cleaning. This can involve the technical process of sanitising data i.e. fixing broken XML, but also the manual deletion of outliers etc.

Are you going to perform statistics to measure the correlation between certain variables?

Or: Are you going to include the data into a machine-learning solution?

In any case, you can plot descriptive statistics to get an impression of the quality of your data, such as:

1. Box-plots to detect outliers
2. Standard deviation, mean, mode, median to assess if it's normally distributed or not
3. More, don't know from the top of my head

Remember that those measures give you an impression of the quality of your data, but even good measurements can come from bad data, and possibly vice versa.

In short, the best way of assessing the quality of your data is to assess the quality of your results, which are acquired generally after several experiments. Only then will you see how well your data is suited for the problem you are trying to solve. That - or you need to get your hands dirty and examine the data qualitatively before you use it.

Data (in assumingly large quantities) are generally used to identify certain patterns within the phenomena described by the data. It is very important to describe your task in terms of a (research) question and a proposed solution or hypothesis. Then, you can assess whether the data is suitable for your particular (research) project. Keep in mind that, theoretically, you can try to answer numerous questions on the same data, or answer the same question on different data sets. The goal is the right balance.

Your question is very broad, and therefore I won't offer very specific answers. You asked for "time-saving tips" but there are many and they depend on context.

Instead, I'll offer a general set of heuristics that are useful most of the time.