3
$\begingroup$

I am new to data science and new to R. I see a lot of resources on data cleaning (i.e., dplyr, tidyr, tidyverse), but not so much on raw value verification. What are the "best practices" I should apply to verify the accuracy of the raw values or detect possible coding errors before cleaning? For example, I have been told you should ensure NAs are true NAs, or ensure there are no spaces in front of numerical or character values after you import data into R.

Thanks in advance.

$\endgroup$
  • 2
    $\begingroup$ This is a seriously critical question, and everybody doing work with data should consider this step. However, any checklist is wholly dependent on the data source, the tools being used, and the robustness of the functions you write/use. For example, read.csv is fine for determining character or numeric, but when just one row is slightly mis-shapen, it can (silently) change the type. readr::read_csv might be more robust to these transient diversions. Some data sources declare "no data" with NA, None, null ... and some tools recognize some, all, or none of those labels. $\endgroup$ – r2evans Jul 11 '18 at 22:20
  • $\begingroup$ Great comment by @r $\endgroup$ – The Lyrist Jul 11 '18 at 23:25
  • $\begingroup$ @r2evans - thank you for the advice; I did not know that about read.csv. It's not so much "R coding techniques" (I clarified my question), but more so general "best practices" or techniques that should be applied to all datasets to detect possible coding errors. $\endgroup$ – DataNoob7 Jul 12 '18 at 0:15
  • $\begingroup$ A lot of it is just experience. Two quotes I like from Oscar Wilde's Lady Windermere's Fan: Experience is the name every one gives to their mistakes, and the follow-on Life would be very dull without them. I must have a lot of "experience" based on my mistakes, and they are occasionally very exciting. $\endgroup$ – r2evans Jul 12 '18 at 0:33
  • $\begingroup$ @r2evans marvelous quotes!! and I wholeheartedly agree. Unfortunately, most organizations aren't like this. I am in an environment where a) I am a noob at datascience (organization is wanting to embrace it thankfully, and I am the only one who has basic knowledge), and b) pressure not to fail is very high. I am quite excited. However, I am trying to avoid the perilous situation say a few years down the line when I have more experience and be like "oh crap, did I do/check "x" in my prior projects". $\endgroup$ – DataNoob7 Jul 12 '18 at 1:14
1
$\begingroup$

UPDDATED with second resource

In general as I understand this is question about 'data quality'. There are lot of manuals, i like suggestions from this one - this is lets say semantic level:

SIX CORE DATA QUALITY DIMENSIONS
    COMPLETENESS 
    UNIQUENESS 
    TIMELINESS
    VALIDITY
    ACCURACY
    CONSISTENCY

This M&E Attribute: Data Quality Assessment checklist is more detailed.

Then, of course, you should go deeper, to the syntax level.

$\endgroup$
  • $\begingroup$ I appreciate the resource, and I will have look. I suspect looking quickly, I am after practices and techniques that assess "validity" and "accuracy". $\endgroup$ – DataNoob7 Jul 12 '18 at 1:36
0
$\begingroup$

Substituting NA values correctly or smartly is also called "imputing" and that can be a very technical topic. See this paper from 2011: MICE: Multivariate Imputation by Chained Equations in R

Another collection of recent material can be found here: A Statistically Sound 'data.frame' Processor/Conditioner • vtreat by R experts Nina Zumel and John Mount.

They focus on preparing data for statistical modeling (which is different from preparing data for, say, an ERP system, or for a scientific computing task).

On that page there are links to technical papers (such as arXiv:1611.09477 stat.AP) and shorter, more readable articles linked from that page where they "[...] demonstrate some of the things that can go wrong with data, and explore ways to address those issues using the R statistical language."

  • Treatment of missing values
  • Treatment of novel levels (new values of categorical variables)
  • Explicit coding of categorical variable levels
  • Treatment of categorical variables with very large numbers of levels
  • Safe processing of “wide data” (data with very many variables)
  • Collaring/Winsorizing of unexpected out of range numeric inputs.

    and many more

To be honest, I think I am an experienced intermediate R programmer and I still struggle with these issues. There is no one-size-fits-it-all checklist that you can work through by crossing things off.

$\endgroup$
  • $\begingroup$ I will check out these resources. It's not so much a checklist, as every dataset is different, it's more so things to be mindful of before data cleaning. $\endgroup$ – DataNoob7 Jul 12 '18 at 10:23
0
$\begingroup$

Once you have the data, the skim function from the skimr package is very handy. It should be all you need to determine if the data were read in correctly or possibly have strange values.

https://github.com/ropensci/skimr

$\endgroup$
  • 2
    $\begingroup$ Someone edited my answer to change data from the plural form to the singular form? Switching back to the correct way. $\endgroup$ – Glen Jul 12 '18 at 16:15
0
$\begingroup$

Another perspective to add to the list:

One of my projects includes a year-long weekly rhythm, where I get a new set of simiarly-structured input files. The process that executes on these input files is rather lengthy, and it might take days for the first symptom of the problem to break out (thereby losing days of processing), so finding violations of my assumptions early is rather important.

If the code were mine, then when I find a problem with the files, the first thing I would do is file an issue/bug, write a test to catch the mistake in generation, fix the bug, then move on. Since the code is not mine, I can only test against the data instead of the code.

So I've adopted using testthat on my input files. (I'm sure other unit-tests could be similarly adapted.) Since the names and structure of each of the files is static-enough for the tests, it's rather straight-forward. Furthermore, testthat provides good mechanisms for automation and returning problems in easily-digested (and if desired verbose) formats for quick identification of problems.

It's not hard to conceive some of these tests up-front: type (int, string, bool), min/max size (exactly 1, no less than 10, no more than 52), and sets (only allow specific values). While I can think of a million other ways that the data might go wrong, I cannot guard against (or check) all of these. So my template starts with the basics: type, size, and sets. If there are obvious boundary values (e.g., always non-negative) then I can incorporate those, but for the most part any test after those defaults are to guard against a failure that I've actually seen (and not just imagined).

(I won't go into how to do write a personal function to adapt these types of tests onto a dataset, the website is pretty robust with examples.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.