There is a quite famous article by H. Wickham, Tidy data, where he defines a certain type of cleaned data and calls it (dataframe-)tidy, and illustrates in on several example using R. At the end, he compares his own definition of dataframe-tidy to other possible ways of achieving tidiness and mentions array-tidy in the following paragraph (but does not give any further explanations or examples):

Fortunately, because there are many efficient tools for working with high-dimensional arrays, even sparse ones, such an array-tidy format is not only likely to be quite compact and efficient, it should also be able to easily connect with the mathematical basis of statistics. This, in fact, is the approach taken by the pandas Python data analysis library (McKinney 2010).

What does array-tidy data mean and why does he imply that this is somehow the default for the pandas library in Python?


TL;DR: Array-tidy is extending tidiness across multiple data frames that are indexed by an additional shared dimension, such as time.

From what I understand, array-tidy and dataframe-tidy refer to the underlying data models that are used to achieve your tidy data. It seems to be a reference to how plyr and pandas differ in their data structure logic

Pandas was developed around the idea of panel data, which is your typical two dimensional data frame of observations and variables, but with samples accumulated over a period of time. Pandas name even stems from this particular type of data (panel data structures, per this McKinney presentation and the pandas docs). The inclusion of a third dimension, thinking about your data over time, seems to have informed some thinking around how pandas would evolve. I recommend checking out the slides (specifically the one titled "The pandas killer feature: indexing").

Relating this back to the Tidy Data paper, creating multiple data frames across an additional dimension (like you would with a Panel in prior versions of pandas) seems to qualify as the "high-dimensional arrays" Wickham discusses in the section you quoted.

I hope this helps!

  • $\begingroup$ Sorry for taking a while to respond, I was out. Wish I could upvote but I dont have enough rep. I have still a number of confusion, unfortunately: 1) So you mean that dataframe-tidy is a just a special case of array-tidy, where we have any array of only two dimensions? $\endgroup$ – l7ll7 Jun 11 '18 at 15:08
  • $\begingroup$ 2) It's not clear to me, why we would actually want to have observations that are not lists of values, but higher dimensional things (such as values at different times) - why would it bad if we, instead of going higher-dimensional stick to lists, and just introduce one more time variable as a column and then for each observation also record at which time it happened? This would seem the straightforward approach to me; I can't see the added value of a multidimensional approach of creating multiple data frames across an additional dimension in this use case. (Sorry for a bit offtopic here.) $\endgroup$ – l7ll7 Jun 11 '18 at 15:09
  • $\begingroup$ 3) looked at the slides you mentioned (thanks for sharing! a lot of that was new to me, since I have never seriously used pandas), but I didn't really understood anything other than: pandas can have a 3 of axes and can deal with them efficiently.Does that mean that pandas are limited to 3-dimensional data - or can you store even higher dimensional data? $\endgroup$ – l7ll7 Jun 11 '18 at 15:09
  • $\begingroup$ 4) On the slide "pandas design philosophy" it says "stop thinking about shapes and start thinking bout indexes". Could you guess what he could have meant? $\endgroup$ – l7ll7 Jun 11 '18 at 15:11
  • $\begingroup$ 1) This seems to be the case $\endgroup$ – tm1212 Jun 11 '18 at 15:46

In general, tidy data refers to a table that has

  • One observation (sample) per row
  • One feature (attribute) per column

Data has historically been recorded like this in ledgers, spreadsheets, csv's etc. Tidy data is easier to interpret and inspect. More importantly, tidy data can be very efficiently processed by modern CPU's/GPU's through vectorization, which is why packages like pandas, excel etc. store data like this.

One interesting fact though, is that neural networks usually take the transpose of a matrix dataset as input - think of this as turning the dataset sideways (so rows are columns and columns are rows) and feeding that into the neural network.

Of course, there are exceptions to this. Think about image data, sound data, some forms of text data; it would not be feasible to store each one of these in a standard row / column table, which is why usually they are stored in other formats (jpeg, txt etc.). Of course, you can transform these types of data so they look more like "tidy-data" for example vectorizing all the pixels of an image, so that you transform a 28x28 image into a vector with 784 columns; now one row is one image and one column is one feature (a pixel location in this case).

  • $\begingroup$ I appreciate the effort you took to write the answer, but I don't think you have quite answered what the difference between dataframe-tidy and array-tidy data is, resp. why this type of tidy data (and not dataframe.tidy data) is used in pandas. $\endgroup$ – l7ll7 Jun 6 '18 at 10:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.