2
$\begingroup$

I have been studying and data science for the past 6 months, however I just came across the principles of tidy data by Hadley Wickham in an article by Jean-Nicholas Hould.

This completely changed my perspective of how I work with data. Not only should I clean the data, but the data should also be formatted correctly. Seems pretty obvious when you think about it, but that is besides the point.

I decided to start applying those principles into my data cleaning workflow, however, I was wondering whether there are times where having tidy data would not be ideal?

When would we not want to "tidy our data"?

Ideally when should we use Tidy Data and when should we avoid it?

Your input to this would be highly appreciated.

$\endgroup$
2
$\begingroup$

In general tidy data is great... but it can quickly become unreasonably large. This is the main reason why I usually try to refactor my data in a tidy format as late as possible in the process.

Example: imagine a dataset containing $N$ instances, with columns feature1 ... featureX and result1... resultY, where the result? columns represent some value based on several methods/parameters. The tidy version would have columns feature1 ... featureX plus method (which takes as values result1,..,resultsY) and of course result for the result value. The tidy version would contain $N \times Y$ instances, i.e. for each instance it repeats the values of the $X$ features $Y$ times. If $X$ is large the dataset is going to be very large in memory (and also stored as a file).

Naturally this gets even worse when you have for example several steps involving various parameters, multiplying the instances every time is not always feasible.

| improve this answer | |
$\endgroup$
  • $\begingroup$ My thought process was the exact same as yours. This got me wondering, when do we draw the line? Is there a standard? Or is this up to one's intuition? (I have also updated the question to reflect that) $\endgroup$ – A Merii Feb 6 at 7:36
  • $\begingroup$ @AMerii I often ask myself the same questions. Usually my choice depends on the data and what I want to do with it: if it's not too large then there's no major problem making it tidy, but otherwise I'll keep with whatever format is convenient. My opinion is that the formalization of "data tidyness" is very insightful and useful in many cases, but it's a theoretical tool which doesn't always fit with practical constraints. $\endgroup$ – Erwan Feb 6 at 17:27
  • $\begingroup$ I completely agree. It would also be nice to know if tidy data should be avoided in machine learning. As sometimes testing and training data need to be formatted in a specific way. I am currently trying to connect with Hadley Wickham on Linked in so that I can get some direct insights from him. Will update the post if I receive a definitive answer from him. $\endgroup$ – A Merii Feb 6 at 17:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.