Add option to choose source of global attributes in MFDataset. #835

Activity

1. jswhit commented on Aug 23, 2018

   

   jswhit

   on Aug 23, 2018

   Last edited by jswhit

   Collaborator

   More actions

   added master_file kwarg arg in pull request #837
2. jswhit commented on Aug 23, 2018

   

   jswhit

   on Aug 23, 2018

   Last edited by jswhit

   Collaborator

   More actions

   @juseg, could you please test the pull request (branch master_file) and let me know if it works for your use case?
3. juseg commented on Aug 24, 2018

   

   juseg

   on Aug 24, 2018

   Author

   More actions

   @jswhit Thanks very much! It works, but the time coordinate is reordered (starting with the master file, and then all other files in order). Potentially this could be solved by either re-ordering subdatasets by filenames, or adding a separate attr_file keyword argument or so.

   PS. I've noticed that the current behaviour of netCDF4 (using global attributes from the first file) is similar to other tools such as ncrcat and cdo cat. I realize this is a pure metadata issue but at least in the context of model simulation chains, apparently concatenation systematically discards history.
4. jswhit commented on Aug 24, 2018

   

   jswhit

   on Aug 24, 2018

   Last edited by jswhit

   Collaborator

   More actions

   @juseg - what do you mean that the time coordinate is re-ordered? If you are aggregating along the time dimension the order of the time coordinate should be set by the order of the files in the files input list. Which file you choose as the master file should not affect this. If it does, that's a bug. It doesn't in my simple test. The master file is only used to define attributes, the variable names and aggregation dimension.
5. juseg commented on Aug 27, 2018

   

   juseg

   on Aug 27, 2018

   Author

   More actions

   @jswhit Sorry that was a mistake, things work exactly as they should. I think we can close this issue.

   import numpy as np
   import netCDF4 as nc
   
   # prepare fake data
   time = np.arange(1000)
   data = time**2
   
   # write to multiple netcdf files
   for i in range(10):
       filename = 'ds' + str(i) + '.nc'
       ds = nc.Dataset(filename, 'w', format='NETCDF4_CLASSIC')
       ds.createDimension('time', None)
       ds.createVariable('time', 'i4', ('time',))
       ds.createVariable('data', 'i4', ('time',))
       ds.variables['time'][:] = time[100*i:100*i+100]
       ds.variables['data'][:] = data[100*i:100*i+100]
       ds.history = 'Created ' + filename + '.'
       ds.close()
   
   # open as mfdataset and assert time is increasing
   ds = nc.MFDataset('ds?.nc', master_file='ds9.nc')
   assert ds.history == 'Created ds9.nc.'
   assert np.all(np.diff(ds.variables['time'][:]) > 0)
   ds.close()

   Many thanks!
6. juseg

   closed this as completedon Aug 27, 2018
7. juseg commented on Aug 27, 2018

   

   juseg

   on Aug 27, 2018

   Author

   More actions

   @jswhit I mean after merging the master_file branch... Sorry for the mess.
8. juseg

   reopened this on Aug 27, 2018
9. jswhit

   closed this as completedon Aug 27, 2018