0
$\begingroup$

With a numpy multidimensional arrays, it is easy to reduce particular axes with functions such as np.mean or np.sum. For example, np.mean(X, axis=2,keepdims=False) will eliminate the second axis.

What if our data is in Pandas long-format DataFrame and we want to reduce a particular variable in a similar fashion?

For example, consider a dataset with four columns: A,B,C, and value. A,B,C specify independent variables, and value specifies a dependent measure. If we wish to reduce the dataframe by eliminating B (i.e., averaging across all levels of B, but not within levels of A and C), we can use df.groupby(['A','C']).mean(). However, this solution becomes very clunky if there are many independent variables that we must enumerate in the groupby call.

Does pandas have an efficient method for achieving numpy-like variable reduction? In principle, one can convert the DataFrame into an xarray, but this seems like an overshoot.

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ why an overshoot? that's exactly what xarray is designed for... $\endgroup$
    – Michael Delgado
    Sep 9, 2021 at 0:32
  • 1
    $\begingroup$ Also Dataframe.to_xarray makes the conversion quite straightforward as well. I'm a little unclear why the groupby solution becomes clunky, you can always programmatically build a list of columns to pass to groupby. $\endgroup$
    – Henry Ecker
    Sep 9, 2021 at 0:33
  • $\begingroup$ @HenryEcker - perhaps a conversion to an xarray, or a coding a function that builds the groupby args are the right solutions. However, since such a reduction is a very common data analytic operation, I was expecting that pandas would have a native way of doing it. $\endgroup$
    – Trisoloriansunscreen
    Sep 9, 2021 at 0:42
  • 1
    $\begingroup$ But you've outlined two of them. You can use groupby for efficient column-wise query-like operations, or if you want the efficiency of n-dimensional arrays, use df.to_xarray(). A DataFrame is not an N-D data structure, so there's no reason to expect it to perform like one. Give xarray a whirl - it's a great package! $\endgroup$
    – Michael Delgado
    Sep 9, 2021 at 0:44

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.