Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I am a CS master student in data mining. My supervisor once told me that before I run any classifier or do anything with a dataset I must fully understand the data and make sure that the data is clean and correct.
My questions:
What are the best practices to understand a dataset (high dimensional with numerical and nominal attributes)?
Practices to make sure the dataset is clean?
Practices to make sure the dataset doesn't have wrong values or so?
There are basic things you can do with any set of data:
Understanding data to identify errors is a whole different ball game, and it is very important.
For instance, you can have a rule that says a serial number must be present in a given data set and that serial number must be alphanumeric with a maximum string length of 255 and a minimum string length of 5.
Looking at the data, you may find one particular serial number value reads "PLEASE ENTER SERIAL" It's perfectly valid, but wrong.
That's kind of an obvious one, but say you're processing stock data and you had a price range for 1000 stocks that was under a dollar. A lot of people would not know that a stock price so low is invalid on certain exchanges and perfectly valid on others. You need knowledge about your data to understand if what you are seeing is problematic or not.
In the real world, you don't always have the luxury of understanding your data intimately.
The way I avoid problems is by leveraging the people around me. For small data sets, I can ask someone to review the data in it's entirety. For large ones, pulling a set of random samples and asking someone to do a sanity check on the data is more appropriate.
Further, questioning the source of the data and how well that data source can be trusted is imperative. I often have multiple conflicting sources of data and we create rules to determine the "source of truth". Sometimes one data set has great data in a given aspect, but other data sets are stronger in other areas.
Manually entered data is usually what I'm most skeptical about, but in some cases it is stronger than anything that can be acquired through automation.
I like @Kallestad answer very much, but I would like to add a meta-step: Make sure that you understand how the data where collected, and what types of constraints there are. I think it is very common to think that there where no non-obvious steps when the data where collected, but this is not the case: Most of the time, some process or indivudal did somethink with the data, and these steps can and will influence the shape of the data.
Two examples: I had a study recently where the data where collected by various con tractors worldwide. I was not at the briefing, so that was opaque to me. Unfortunately, the measurements where off for some parts of france: People all liked ice cram, but we expected a random distribution. There was no obvious reason for this uniformity, so I began to hunt the errors. When I queried the contractors, one had misunderstood the briefing and selected only ice-cream lovers from his database.
The second error was more challenging: When doing some geographic analysis, I found that a lot of people had extremely large movement patterns, which suggested that a lot of them traveled from Munich to Hamburg in minutes. When I spoke with ppeople upstream, they found a subtle bug in their data aggregation software, which was unnoticed before.
Conclusions:
I usually take a two-step approach
compute univariate (variable by variable) summary statistics such as mean, range, variance, number of missing, cardinality, etc. for each variable and look for oddities (e.g. range not plausible given the meaning of the variable). Plot histograms for those odd variables.
split the data into manageable subsets (choose a meaningful variable and split the data according to it e.g. all positive examples, and all negative) and explore them visually (e.g. with ggobi ). Especially use tools like brushing and scatter plots to understand how variables are linked together.
And when you start building models, make sure to plot the residuals, looking for extreme errors that might be due to an outlier, or look at the confusion matrix and make sure it is balanced. Use k-fold cross validation to optimize your models and look at the variance of the training error for each fold, if one fold performs much worse than the others, it may contain outliers.
Below you can find a copy of my answer to a related (however, focused on data cleaning aspect) question here on Data Science StackExchange (https://datascience.stackexchange.com/a/722/2452), provided in its entirety for readers' convenience. I believe that it partially answers your question as well and hope it is helpful. While the answer is focused on R ecosystem, similar packages and/or libraries can be found for other data analysis environments. Moreover, while the two cited papers on data preparation also contain examples in R, these papers present general workflow (framework) and best practices that are applicable to any data analysis environment.
R contains some standard functions for data manipulation, which can be used for data cleaning, in its base package (gsub, transform, etc.), as well as in various third-party packages, such as stringr, reshape, reshape2, and plyr. Examples and best practices of usage for these packages and their functions are described in the following paper: http://vita.had.co.nz/papers/tidy-data.pdf.
Additionally, R offers some packages specifically focused on data cleaning and transformation:
A comprehensive and coherent approach to data cleaning in R, including examples and use of editrules and deducorrect packages, as well as a description of workflow (framework) of data cleaning in R, is presented in the following paper, which I highly recommend: http://cran.r-project.org/doc/contrib/de_Jonge+van_der_Loo-Introduction_to_data_cleaning_with_R.pdf.
Folks here stated great steps, but I think there are great information at the following link what I do when I get a new data set as told through tweets, It sums up the steps the folks tweeted answering the great @hmason question "Data people: What is the very first thing you do when you get your hands on a new data set?"
Hope it will be useful.
I'll add one thing- if possible, do a reasonableness check by comparing you data against some other source. It seems that whenever I fail to do this, I get burnt:(
