Lazy concatenation of arrays #4628

Activity

1. shoyer commented on May 24, 2021

   

   shoyer

   on May 24, 2021

   Member

   More actions

   If you write write something like xarray.concat(..., data_vars='minimal', coords='minimal'), dask should entirely lazy -- the non-laziness only happens with the default value of coords='different'.

   But I agree, it would be nice if Xarray's internal lazy indexing machinery supported concatenation. It currently does not.
2. dcherian

   added

   topic-combinecombine/concat/mergecombine/concat/merge

   on Jul 8, 2021
3. LunarLanding commented on Nov 25, 2021

   

   LunarLanding

   on Nov 25, 2021

   More actions

   Any pointers regarding where to start / modules involved to implement this? I would like to have a try.
4. dcherian

   added

   topic-lazy array

   on Apr 19, 2022
5. dcherian commented on May 10, 2022

   

   dcherian

   on May 10, 2022

   Contributor

   More actions

   From @rabernat in #6588:

   Right now, if I want to concatenate multiple datasets (e.g. as in open_mfdataset), I have two options:

   • Eagerly load the data as numpy arrays ➡️ xarray will dispatch to np.concatenate
   • Chunk each dataset ➡️ xarray will dispatch to dask.array.concatenate

   In pseudocode:

   ds1 = xr.open_dataset("some_big_lazy_source_1.nc")
   ds2 = xr.open_dataset("some_big_lazy_source_2.nc")
   item1 = ds1.foo[0, 0, 0]  # lazily access a single item
   ds = xr.concat([ds1.chunk(), ds2.chunk()], "time")  # only way to lazily concat
   # trying to access the same item will now trigger loading of all of ds1
   item1 = ds.foo[0, 0, 0]
   # yes I could use different chunks, but the point is that I should not have to 
   # arbitrarily choose chunks to make this work
   

   However, I am increasingly encountering scenarios where I would like to lazily concatenate datasets (without loading into memory), but also without the requirement of using dask. This would be useful, for example, for creating composite datasets that point back to an OpenDAP server, preserving the possibility of granular lazy access to any array element without the requirement of arbitrary chunking at an intermediate stage.

   Describe the solution you'd like

   I propose to extend our LazilyIndexedArray classes to support simple concatenation and stacking. The result of applying concat to such arrays will be a new LazilyIndexedArray that wraps the underlying arrays into a single object.

   The main difficulty in implementing this will probably be with indexing: the concatenated array will need to understand how to map global indexes to the underling individual array indexes. That is a little tricky but eminently solvable.

   Describe alternatives you've considered

   The alternative is to structure your code in a way that avoids needing to lazily concatenate arrays. That is what we do now. It is not optimal.

   ❤️React with ❤️2Reacted by Vincent Landau and Tom Nicholas
6. nbren12 commented on May 10, 2022

   

   nbren12

   on May 10, 2022

   ContributorAuthor

   More actions

   @rabernat It seems that great minds think alike ;)

   😄React with 😄2Reacted by Ryan Abernathey and Deepak Cherian
7. rabernat commented on May 10, 2022

   

   rabernat

   on May 10, 2022

   Last edited by rabernat

   Contributor

   More actions

   Any pointers regarding where to start / modules involved to implement this? I would like to have a try.

   The starting point would be to look at the code in indexing.py and try to understand how lazy indexing works.

   In particular, look at

   xarray/xarray/core/indexing.py

   Lines 465 to 470 in 3920c48

   	class LazilyIndexedArray(ExplicitlyIndexedNDArrayMixin):
   	"""Wrap an array to make basic and outer indexing lazy."""
   	__slots__ = ("array", "key")
   	def __init__(self, array, key=None):

   Then you may want to try writing a class that looks like

    class LazilyConcatenatedArray:  # have to decide what to inherit from 
     
       def __init__(self, *arrays: LazilyIndexedArray, concat_axis=0):
           # figure out what you need to keep track of
   
       @property
       def shape(self):
           # figure out how to determine the total shape
   
       def __getitem__(self, indexer) -> LazilyIndexedArray:
           # figure out how to map an indexer to the right piece of data
8. ilan-gold commented on Sep 23, 2024

   

   ilan-gold

   on Sep 23, 2024

   Last edited by ilan-gold

   Contributor

   More actions

   I would +1 your great proposal @dcherian . My use-case would be the extension arrays - being able to concat them without dask is basically a must for us.

   Can't do this without virtual concat machinery (#4628) which someone decided to implement elsewhere 🙄 ;)

   From #9038 (comment) - what is this referring to?

   In short, this is not implemented here: scverse/anndata#1247 (comment) . It is implemented for non-lazy categoricals but not lazy ones
9. TheGreatAlgo commented on Nov 12, 2025

   

   TheGreatAlgo

   on Nov 12, 2025

   More actions

   This would be really helpful for us as it relates to our massive weather datasets (~3TB in size). We have zarrs of weather data that has different attributes such as in the case of ERA5, finalized or unfinalized data. Finalized meaning no more future changes and unfinalized meaning it may still change. However for someone wanting to run an analysis on a specific location that stretches across the datasets they would need to query individually and concat together.

   ERA5 finalized may end on June for example and then unfinalized they can get data up to November. If you want the full historical weather you need to query both separately and then merge them together.

   Our idea was just to return them both concated together (3TB for finalized reaching back to 1940s and 8GB in this example that might reach back a couple months) and then the user can just query against the whole xarray dataset like normal without much effort.

   I have tried the minimal trick but still seems to silently fail for me after an uptick in cpu usage.