ValueError when trying to encode time variable in a NetCDF file with CF convensions #3739

Activity

1. rabernat commented on Jan 31, 2020

   

   rabernat

   on Jan 31, 2020

   Contributor

   More actions

   Hi @avatar101 - thanks for your issue!

   I couldn't fully reproduce your example, since it references files that I don't have access to. For the future, please consider trying to create a "Minimal, Complete and Verifiable Example" (MCVE): (http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports)

   In the meantime, I do have a suggestion you could try. Instead of

   ds_test.time.attrs['units'] = "Seconds since 1970-01-01 00:00:00"

   try

   ds_test.time.encoding['units'] = "Seconds since 1970-01-01 00:00:00"

   The reason is that, since you have created your time coordinate using pd.date_range, which returns a datetime64 dtype, that variable is already considered to be encoded.
2. avatar101 commented on Jan 31, 2020

   

   avatar101

   on Jan 31, 2020

   Author

   More actions

   Hi Ryan, thanks for your reply. Apologies for not creating a reproducible problem earlier as the files weren't created by xarray routine. Please find my attempt at reproducing the problem below:

   Minimum steps to reproduce the error

   import numpy as np
   import xarray as xr
   import pandas as pd
   
   data1 = np.ones(shape=(1, 181, 360))
   lats=np.arange(-90,91, 1)
   lons=np.arange(-180,180,1)
   time1 = np.array([0])
   
   # creating the first dataset
   da_1 = xr.DataArray(data1, coords=[time1, lats, lons], dims=['time', 'lats', 'lons'])
   da_1.time.attrs['units'] = "hours since 1988-01-01 00:00:00"
   da_1.time.attrs['calendar'] = "proleptic_gregorian"
   da_1.time.attrs['standard_name'] = "time"
   ds_1 = xr.Dataset({'V':da_1})
   ds_1.attrs['Conventions'] = 'CF'
   ds_1.to_netcdf('ds_1.nc', encoding=None)
   
   # creating second test dataset
   time2=np.array([6])  # wrong time value
   da_2 = xr.DataArray(data1, coords=[time2, lats, lons], dims=['time', 'lats', 'lons'])
   da_2.time.attrs['units'] = "hours since 1988-01-01 06:00:00"
   da_2.time.attrs['calendar'] = "proleptic_gregorian"
   da_2.time.attrs['standard_name'] = "time"
   
   ds_2 = xr.Dataset({'V':da_2})
   ds_2.attrs['Conventions'] = 'CF'
   # saving it with wrong time value
   ds_2.to_netcdf('ds_2.nc', encoding=None)
   
   
   # Reading the 2 files and concatenating them
   files = ['/path/to/ds_1.nc', '/path/to/ds_2.nc']
   
   ds_test = xr.open_mfdataset(files, combine='nested', concat_dim='time', decode_cf=False)
   yr = 1988 # year
   dates = pd.date_range(start=(yr), end=str(yr+1), freq='6H', closed='left')
   ds_test.time.values=dates[:2] # fixing the time values 
   ds_test.time.attrs['units'] = "Seconds since 1970-01-01 00:00:00"  #required encoding
   ds_test.to_netcdf('ds_1_2.nc')  # gives the same error

   ValueError: failed to prevent overwriting existing key units in attrs on variable 'time'. This is probably an encoding field used by xarray to describe how a variable is serialized. To proceed, remove this key from the variable's attributes manually.

   I've also mentioned your suggestion earlier in the original post. It also gives the same error message
   Please find the following reproducible steps incorporating your suggestion.

   Trying time encoding solution

   # Reading the files 
   files = ['/path/to/ds_1.nc', '/path/to/ds_2.nc']
   
   ds_test = xr.open_mfdataset(files, combine='nested', concat_dim='time', decode_cf=False)
   yr = 1988 # year
   dates = pd.date_range(start=(yr), end=str(yr+1), freq='6H', closed='left')
   ds_test.time.values=dates[:2] # fixing the time values 
   # encoding try
   ds_test.time.encoding['units'] = "Seconds since 1970-01-01 00:00:00"
   ds_test.to_netcdf('ds_1_2.nc')  # gives same error
   

   ValueError: failed to prevent overwriting existing key calendar in attrs on variable 'time'. This is probably an encoding field used by xarray to describe how a variable is serialized. To proceed, remove this key from the variable's attributes manually.
3. Chan-Jer commented on Feb 17, 2020

   

   Chan-Jer

   on Feb 17, 2020

   More actions

   hey,avatar, thanks for your examples. I'm wondering that problem has figured out or not . I have met the same problem.
   I'm appreciated for any help.
4. mathause commented on Feb 17, 2020

   

   mathause

   on Feb 17, 2020

   Collaborator

   More actions

   The following works (using assign_coords() instead of time.values=). I think you also forgot a str in date_range.

   import numpy as np
   import xarray as xr
   import pandas as pd
   
   files = ['ds_1.nc', 'ds_2.nc']
   
   ds_test = xr.open_mfdataset(files, combine='nested', concat_dim='time', decode_cf=False)
   yr = 1988 # year
   dates = pd.date_range(start=str(yr), end=str(yr+1), freq='6H', closed='left')
   ds_test.assign_coords(time=dates[:2])
   ds_test.time.encoding['units'] = "seconds since 1970-01-01 00:00:00"
   ds_test.time.encoding['calendar'] = "proleptic_gregorian"
   ds_test.to_netcdf('ds_1_2.nc')

   The following works as well:

   ds_test = xr.open_mfdataset(files, combine='nested', concat_dim='time')
   ds_test.time.encoding['units'] = "seconds since 1970-01-01 00:00:00"
   ds_test.to_netcdf('ds_1_2.nc')

   ---

   However, what does indeed not work is the following

   ds_test = xr.open_mfdataset(files, combine='nested', concat_dim='time')
   ds_test.time.attrs['units'] = "seconds since 1970-01-01 00:00:00"
   ds_test.to_netcdf('ds_1_2.nc')

   which I don't entirely understand, because ds_test.time.encoding is empty. So maybe there is an encoding hidden somewhere, but I couldn't find it.
5. avatar101 commented on Feb 23, 2020

   

   avatar101

   on Feb 23, 2020

   Author

   More actions

   @mathause

   thanks for your suggestions. Your first solution works fine for correcting the time data stored in the array. I also don't understand why ds_test.time.encoding is empty and yet, its the reason for an error while saving it. Maybe its a bug?

   @Chan-Jer Another work around which I used was to set the correct time value using cdo's settime function.
6. bonnland commented on Feb 23, 2021

   

   bonnland

   on Feb 23, 2021

   More actions

   I got a similar error message when opening a Zarr store with datetime64 time values, where I tried to set the "calendar" attribute on the time axis (the attribute was unset in the original store). I've found some xarray code that appears to treat the "calendar" and "units" time attributes as special, and it essentially prevents users from setting or changing these values, even when those values are not present:

   From xarray/coding/times.py:

   class CFDatetimeCoder(VariableCoder):
       def __init__(self, use_cftime=None):
           self.use_cftime = use_cftime
   
       def encode(self, variable, name=None):
           dims, data, attrs, encoding = unpack_for_encoding(variable)
           if np.issubdtype(data.dtype, np.datetime64) or contains_cftime_datetimes(
               variable
           ):
               (data, units, calendar) = encode_cf_datetime(
                   data, encoding.pop("units", None), encoding.pop("calendar", None)
               )
               safe_setitem(attrs, "units", units, name=name)
               safe_setitem(attrs, "calendar", calendar, name=name)
   
           return Variable(dims, data, attrs, encoding)
   
7. dcherian

   added

   topic-CF conventions

   on Feb 23, 2021
8. kmuehlbauer commented on Sep 13, 2023

   

   kmuehlbauer

   on Sep 13, 2023

   Contributor

   More actions

   The initial issue seems to be sorted out, by overwriting/setting the needed keys in encoding instead of attrs. If you still have issues with this, please open a new issue with MCVE. Thanks!
9. kmuehlbauer

   closed this as completedon Sep 13, 2023