When should we use the principles of tidy data by Hadley Whickham, and when should we avoid using it?

Question

I have been studying and data science for the past 6 months, however I just came across the principles of tidy data by Hadley Wickham in an article by Jean-Nicholas Hould.

This completely changed my perspective of how I work with data. Not only should I clean the data, but the data should also be formatted correctly. Seems pretty obvious when you think about it, but that is besides the point.

I decided to start applying those principles into my data cleaning workflow, however, I was wondering whether there are times where having tidy data would not be ideal?

When would we not want to "tidy our data"?

Ideally when should we use Tidy Data and when should we avoid it?

Your input to this would be highly appreciated.

Answer 1

In general tidy data is great... but it can quickly become unreasonably large. This is the main reason why I usually try to refactor my data in a tidy format as late as possible in the process.

Example: imagine a dataset containing $N$ instances, with columns feature1 ... featureX and result1... resultY, where the result? columns represent some value based on several methods/parameters. The tidy version would have columns feature1 ... featureX plus method (which takes as values result1,..,resultsY) and of course result for the result value. The tidy version would contain $N \times Y$ instances, i.e. for each instance it repeats the values of the $X$ features $Y$ times. If $X$ is large the dataset is going to be very large in memory (and also stored as a file).

Naturally this gets even worse when you have for example several steps involving various parameters, multiplying the instances every time is not always feasible.