Tidy data in panda dataframe

Question

There is a quite famous article by H. Wickham, Tidy data, where he defines a certain type of cleaned data and calls it (dataframe-)tidy, and illustrates in on several example using R. At the end, he compares his own definition of dataframe-tidy to other possible ways of achieving tidiness and mentions array-tidy in the following paragraph (but does not give any further explanations or examples):

Fortunately, because there are many efficient tools for working with high-dimensional arrays, even sparse ones, such an array-tidy format is not only likely to be quite compact and efficient, it should also be able to easily connect with the mathematical basis of statistics. This, in fact, is the approach taken by the pandas Python data analysis library (McKinney 2010).

What does array-tidy data mean and why does he imply that this is somehow the default for the pandas library in Python?

Answer 1

TL;DR: Array-tidy is extending tidiness across multiple data frames that are indexed by an additional shared dimension, such as time.

From what I understand, array-tidy and dataframe-tidy refer to the underlying data models that are used to achieve your tidy data. It seems to be a reference to how plyr and pandas differ in their data structure logic

Pandas was developed around the idea of panel data, which is your typical two dimensional data frame of observations and variables, but with samples accumulated over a period of time. Pandas name even stems from this particular type of data (panel data structures, per this McKinney presentation and the pandas docs). The inclusion of a third dimension, thinking about your data over time, seems to have informed some thinking around how pandas would evolve. I recommend checking out the slides (specifically the one titled "The pandas killer feature: indexing").

Relating this back to the Tidy Data paper, creating multiple data frames across an additional dimension (like you would with a Panel in prior versions of pandas) seems to qualify as the "high-dimensional arrays" Wickham discusses in the section you quoted.

I hope this helps!

Answer 2

In general, tidy data refers to a table that has

• One observation (sample) per row
• One feature (attribute) per column

Data has historically been recorded like this in ledgers, spreadsheets, csv's etc. Tidy data is easier to interpret and inspect. More importantly, tidy data can be very efficiently processed by modern CPU's/GPU's through vectorization, which is why packages like pandas, excel etc. store data like this.

One interesting fact though, is that neural networks usually take the transpose of a matrix dataset as input - think of this as turning the dataset sideways (so rows are columns and columns are rows) and feeding that into the neural network.

Of course, there are exceptions to this. Think about image data, sound data, some forms of text data; it would not be feasible to store each one of these in a standard row / column table, which is why usually they are stored in other formats (jpeg, txt etc.). Of course, you can transform these types of data so they look more like "tidy-data" for example vectorizing all the pixels of an image, so that you transform a 28x28 image into a vector with 784 columns; now one row is one image and one column is one feature (a pixel location in this case).