With a numpy multidimensional arrays, it is easy to reduce particular axes with functions such as np.mean or np.sum. For example, np.mean(X, axis=2,keepdims=False) will eliminate the second axis.

What if our data is in Pandas long-format DataFrame and we want to reduce a particular variable in a similar fashion?

For example, consider a dataset with four columns: A,B,C, and value. A,B,C specify independent variables, and value specifies a dependent measure. If we wish to reduce the dataframe by eliminating B (i.e., averaging across all levels of B, but not within levels of A and C), we can use df.groupby(['A','C']).mean(). However, this solution becomes very clunky if there are many independent variables that we must enumerate in the groupby call.

Does pandas have an efficient method for achieving numpy-like variable reduction? In principle, one can convert the DataFrame into an xarray, but this seems like an overshoot.

  • 1
    $\begingroup$ why an overshoot? that's exactly what xarray is designed for... $\endgroup$
    – Michael Delgado
    Sep 9, 2021 at 0:32
  • 1
    $\begingroup$ Also Dataframe.to_xarray makes the conversion quite straightforward as well. I'm a little unclear why the groupby solution becomes clunky, you can always programmatically build a list of columns to pass to groupby. $\endgroup$
    – Henry Ecker
    Sep 9, 2021 at 0:33
  • $\begingroup$ @HenryEcker - perhaps a conversion to an xarray, or a coding a function that builds the groupby args are the right solutions. However, since such a reduction is a very common data analytic operation, I was expecting that pandas would have a native way of doing it. $\endgroup$ Sep 9, 2021 at 0:42
  • 1
    $\begingroup$ But you've outlined two of them. You can use groupby for efficient column-wise query-like operations, or if you want the efficiency of n-dimensional arrays, use df.to_xarray(). A DataFrame is not an N-D data structure, so there's no reason to expect it to perform like one. Give xarray a whirl - it's a great package! $\endgroup$
    – Michael Delgado
    Sep 9, 2021 at 0:44


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.