Support group_over

Multi-dimensional grouped operations should be relatively straightforward -- the main complexity will be writing an N-dimensional concat that doesn't involve repetitively copying data.

The idea with group_over would be to support groupby operations that act on a single element from each of the given groups, rather than the unique values. For example, ds.group_over(['lat', 'lon']) would let you iterate over or apply to 2D slices of ds, no matter how many dimensions it has.

Roughly speaking (it's a little more complex for the case of non-dimension variables), ds.group_over(dims) would get translated into ds.groupby([d for d in ds.dims if d not in dims]).

Related: #266

@shoyer

@shoyer -

I want to look into putting a PR together for this. I'm looking for the same functionality that you get with a pandas Series or DataFrame:

data.groupby([lambda x: x.hour, lambda x: x.timetuple().tm_yday]).mean()

The motivation comes in making a Hovmoller diagram. What we need is this functionality:

da.groupby(['time.hour', 'time.dayofyear']).mean().plot()

If you can point me in the right direction, I'll see if I can put something together.

@jhamman

@jhamman For your use case, both hour and dayofyear are along the time dimension, so arguably the result should be 1D with a MultiIndex instead of 2D. So it might make more sense to start with that, and then layer on stack/unstack or pivot functionality.

I guess there are two related use cases here:

Multiple groupby arguments along a single dimension (pandas does this one already)
Multiple groupby arguments along different dimensions (pandas doesn't do this one).

Agreed, we have two use cases here.

For (1), can we just use the pandas grouping infrastructure. We just need to allow xray.DataArray.groupby to support an iterable and pandas.Grouper objects. I personally don't like the MultiIndex format and prefer to unstack the grouper operations when possible. In xray, I think we can justify going that route since we support N-D labeled dimensions much better than pandas.

For (2), I'll need to think a bit more about how this would work. Do we add a groupby method to DataArrayGroupBy? That sounds messy. Maybe we need to write a N-D grouper object?

For (2) I think it makes sense to extend the existing groupby to deal with multiple dimensions. Ie, let it take an iterable of dimension names.

>>> darray.groupby(['lat', 'lon'])

Then we'd have something similar to the SQL groupby, which is a good thing.

By the way, in #527 we were considering using this approach to make the faceted plots on both rows and columns.

In case it is of interest to anyone, the snippet below is a temporary and quite dirty solution I've used to do a multi-dimensional groupby...

It runs nested groupby-apply operations over each given dimension until no further grouping needs to be done, then applies the given function "apply_fn"

def nested_groupby_apply(dataarray, groupby, apply_fn):
    if len(groupby) == 1:
        return dataarray.groupby(groupby[0]).apply(apply_fn)
    else:
        return dataarray.groupby(groupby[0]).apply(nested_groupby_apply, groupby = groupby[1:], apply_fn = apply_fn)

Obviously performance can potentially be quite poor. Passing the dimensions to group over in order of increasing length will reduce your cost a little.

Is use case 1 (Multiple groupby arguments along a single dimension) being held back for use case 2 (Multiple groupby arguments along different dimensions)? Use case 1 would be very useful by itself.

Is use case 1 (Multiple groupby arguments along a single dimension) being held back for use case 2 (Multiple groupby arguments along different dimensions)? Use case 1 would be very useful by itself.

No, I think the biggest issue is that grouping variables into a MultiIndex on the result sort of works (with the current PR #924), but it's very easy to end up with weird conflicts between coordinates / MultiIndex levels that are hard to resolve right now within the xarray data model. Probably it would be best to resolve #1603 first, which will make this much easier.

In order to maintain a list of currently relevant issues, we mark issues as stale after a period of inactivity

If this issue remains relevant, please comment here or remove the stale label; otherwise it will be marked as closed automatically

Still relevant.

still relevant, also for me ... I just wanted to group by half hours, for which I'd need access to .groupby(['time.hour','time.minutes'])

Still relevant, would like to be able to group by multiple variables along a single dimension.

I have this almost ready in flox (needs more tests). So we should be able to do this soon.

In the mean time note that we can view grouping over multiple variables as a "factorization" (group identification) problem for aggregations. That means you can

use pd.factorize, pd.cut, np.searchsorted or np.bincount to convert each by variable to an integer code,
then use np.ravel_multi_index to combine the codes to a single variable idx
Group by idx and accumulate
use np.unravel_index (or just a simple np.reshape) to convert the single grouped dimension to a multiple dimensions.
Construct output coordinate arrays.

GroupBy(multiple groupers)

Navigation Menu

Repository navigation

Support group_over #324

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Support group_over #324

Description

Activity

jhamman commented on Aug 16, 2015

shoyer commented on Aug 17, 2015

jhamman commented on Aug 17, 2015

clarkfitzg commented on Aug 17, 2015

JonAnCla commented on Dec 7, 2016

jjpr-mit commented on Oct 16, 2017

shoyer commented on Oct 16, 2017

stale commented on Sep 16, 2019

shoyer commented on Sep 16, 2019

matthiasdemuzere commented on Feb 18, 2021

alimanfoo commented on Feb 28, 2022

dcherian commented on Feb 28, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions