User-configurable datamode for UFS

@uturuncoglu

Current data component datamodes explicitly define the variables in streams and to advertise.
To map given input data to particular export variables, this is solid.

Once there can be different input and export variables involving calculations and unit conversions that depend on values in the data, this gets complicated. Especially to know if datamode is behaving properly (#396) across combination of options.

If a user has known input data that should just pass through to export state, a generic datamode may help #411 and is like if atm/ocn/ice/... had inline-like datamode too. But that's just a pass-through copy.

From ERA5 discussions @uturuncoglu @raphaeldussin @sanAkel @sdurski , it seems you want

Convert precipitation in units of meters (cumulative meters of rain per hour), rather than kg/m^2/s
Convert radiation in units of Joules (cumulative energy per hour), rather than W/m^2 flux
Partition net shortwave radiation into bands
Partition net precipitation into rain and snow

I think we can have configurable variables and specific configurable calculations/conversions.

Chatting with Gemini AI, mapped out an architecture for modular, unified datamode that addresses this by separating concerns of stream ingestion, export variable names, and derived calculations.

Design elements are:

Dynamic Ingestion (Zero-Boilerplate Pass-through): Instead of hardcoding pointer assignments for every possible variable (e.g., Sa_z(:) = strm_Sa_z(:)), the datamode dynamically parses the ESMF stream configuration. It allocates a mapping array and uses a single loop with native CDEPS APIs (shr_strdata_get_stream_pointer and dshr_state_getfldptr) to map whatever streams the user provides directly to the export state.
Chained Post-Processing: We introduce a new space-separated string to the namelist (calc_opts, namelist configurable). A calculation driver parses this string and executes isolated subroutines (like calc_partition_precip_freezing or calc_convert_rad_accum_to_flux) that mutate the variables in-place directly on the exportState.

Key Benefits of this Approach:

No More Datamode Proliferation: We stop writing custom datamodes (like GEFS, JRA55, etc.) for every new data source. The user defines the variables in their config, and the code handles the mapping dynamically.
Fail-Fast Validation: By leveraging requirePointer=.true. in the calculation subroutines, if a user requests a calculation in calc_opts but forgets to provide the prerequisite data stream, CDEPS natively halts the MPI ranks with a clear error before any math is executed.
Highly Extensible: Adding a new unit conversion or partitioning scheme in the future simply requires dropping a single isolated subroutine into the driver, rather than untangling inline logic.

Quick draft is like:

!===============================================================================
!
!  Module: cdeps_datamode_unified_mod
!
!  Description:
!    Unified datamode for CDEPS. Replaces hardcoded datamodes with a dynamic, 
!    configuration-driven approach. 
!    
!    Phase 1: Automatically maps any variable defined in the ESMF config stream 
!             directly to the export state.
!    Phase 2: Executes user-configurable, chained derived-variable calculations 
!             driven by the calc_opts string passed from the NUOPC cap.
!
!===============================================================================
module cdeps_datamode_unified_mod
  
  use ESMF
  use NUOPC
  
  ! CDEPS-native imports
  use shr_kind_mod,    only: r8 => shr_kind_r8
  use shr_strdata_mod, only: shr_strdata_type, shr_strdata_get_stream_pointer
  use dshr_state_mod,  only: dshr_state_getfldptr
  use shr_log_mod,     only: shr_log_error
  use dshr_utils_mod,  only: ChkErr
  use dshr_stream_mod, only: dshr_stream_type, shr_stream_init_from_esmfconfig

  implicit none
  private
  
  public :: DatamodeUnifiedInit
  public :: DatamodeUnifiedRun
  public :: unified_datamode_type

  character(len=*), parameter :: u_FILE_u = 'cdeps_datamode_unified_mod.F90'

  !> \brief Maps configuration string to persistent memory pointers
  type :: unified_var_map
     character(len=64) :: var_name
     real(r8), pointer :: ptr_strm(:) => null()
     real(r8), pointer :: ptr_exp(:)  => null()
  end type unified_var_map

  !> \brief Master state for the unified datamode
  type :: unified_datamode_type
     type(unified_var_map), allocatable :: var_maps(:)
  end type unified_datamode_type

contains

  !=============================================================================
  ! \brief Parse ESMF config for stream variables and dynamically build map
  !=============================================================================
  subroutine DatamodeUnifiedInit(config, unified_state, rc)
    type(ESMF_Config),             intent(in)    :: config
    type(unified_datamode_type),   intent(inout) :: unified_state
    integer,                       intent(out)   :: rc
    
    character(len=*), parameter :: subName = 'DatamodeUnifiedInit: '
    type(dshr_stream_type), allocatable :: streams(:)
    integer :: i, j

    rc = ESMF_SUCCESS

    ! Parse ESMF Config for 1-to-1 Stream Variables (Dynamically Appended)
    call shr_stream_init_from_esmfconfig(streams, config, rc=rc)
    if (ChkErr(rc,__LINE__,u_FILE_u)) return

    if (allocated(streams)) then
      do i = 1, size(streams)
        do j = 1, streams(i)%nFlds
          ! Dynamically grow the var_maps array using move_alloc
          call append_var_map(unified_state%var_maps, streams(i)%fldName(j))
        end do
      end do
      deallocate(streams)
    end if

  end subroutine DatamodeUnifiedInit


  !=============================================================================
  ! \brief Maps raw variables, then chains calculations driven by calc_opts
  !=============================================================================
  subroutine DatamodeUnifiedRun(sdat, exportState, unified_state, calc_opts, rc)
    type(shr_strdata_type),        intent(inout) :: sdat
    type(ESMF_State),              intent(inout) :: exportState
    type(unified_datamode_type),   intent(inout) :: unified_state
    character(len=*),              intent(in)    :: calc_opts
    integer,                       intent(out)   :: rc
    
    character(len=*), parameter :: subName = 'DatamodeUnifiedRun: '
    integer :: i

    rc = ESMF_SUCCESS

    ! -------------------------------------------------------------------------
    ! Phase 1: Ingestion (Stream -> Export State)
    ! -------------------------------------------------------------------------
    if (allocated(unified_state%var_maps)) then
      do i = 1, size(unified_state%var_maps)
        
        call shr_strdata_get_stream_pointer(sdat, trim(unified_state%var_maps(i)%var_name), &
             unified_state%var_maps(i)%ptr_strm, requirePointer=.true., &
             errmsg=trim(subName)//'ERROR: stream pointer missing for '//trim(unified_state%var_maps(i)%var_name), &
             rc=rc)
        if (ChkErr(rc,__LINE__,u_FILE_u)) return
        
        call dshr_state_getfldptr(exportState, trim(unified_state%var_maps(i)%var_name), &
             fldptr1=unified_state%var_maps(i)%ptr_exp, rc=rc)
        if (ChkErr(rc,__LINE__,u_FILE_u)) return
        
        unified_state%var_maps(i)%ptr_exp(:) = unified_state%var_maps(i)%ptr_strm(:)
        
      end do
    end if

    ! -------------------------------------------------------------------------
    ! Phase 2: Chained Calculations Driver
    ! -------------------------------------------------------------------------
    ! Only enter the driver if there are options requested to save overhead
    if (len_trim(calc_opts) > 0) then
      call DatamodeUnifiedCalcDriver(exportState, calc_opts, rc)
      if (ChkErr(rc,__LINE__,u_FILE_u)) return
    end if
    
  end subroutine DatamodeUnifiedRun


  !=============================================================================
  ! \brief Subroutine to dynamically append variables (Fortran 2003 move_alloc)
  !=============================================================================
  subroutine append_var_map(var_maps, new_var_name)
    type(unified_var_map), allocatable, intent(inout) :: var_maps(:)
    character(len=*),                   intent(in)    :: new_var_name
    
    type(unified_var_map), allocatable :: temp_maps(:)
    integer :: current_size

    if (allocated(var_maps)) then
      current_size = size(var_maps)
      allocate(temp_maps(current_size + 1))
      temp_maps(1:current_size) = var_maps
      temp_maps(current_size + 1)%var_name = trim(new_var_name)
      call move_alloc(from=temp_maps, to=var_maps)
    else
      allocate(var_maps(1))
      var_maps(1)%var_name = trim(new_var_name)
    end if
  end subroutine append_var_map


  !=============================================================================
  ! \brief Driver subroutine to route execution based on calc_opts string
  !=============================================================================
  subroutine DatamodeUnifiedCalcDriver(exportState, calc_opts, rc)
    type(ESMF_State), intent(inout) :: exportState
    character(len=*), intent(in)    :: calc_opts
    integer,          intent(out)   :: rc
    
    rc = ESMF_SUCCESS

    ! 1. Convert cumulative precipitation (m) to rate (kg/m^2/s)
    if (index(calc_opts, 'convert_precip_accum_to_rate') > 0) then
      call calc_convert_precip_accum_to_rate(exportState, rc)
      if (ChkErr(rc,__LINE__,u_FILE_u)) return
    end if

    ! 2. Convert cumulative radiation (J/m^2) to flux (W/m^2)
    if (index(calc_opts, 'convert_rad_accum_to_flux') > 0) then
      call calc_convert_rad_accum_to_flux(exportState, rc)
      if (ChkErr(rc,__LINE__,u_FILE_u)) return
    end if

    ! 3. Partition net shortwave radiation into bands
    if (index(calc_opts, 'partition_sw_4band') > 0) then
      call calc_partition_sw_4band(exportState, rc)
      if (ChkErr(rc,__LINE__,u_FILE_u)) return
    end if

    ! 4. Partition precipitation into rain and snow
    if (index(calc_opts, 'partition_precip_freezing') > 0) then
      call calc_partition_precip_freezing(exportState, rc)
      if (ChkErr(rc,__LINE__,u_FILE_u)) return
    end if

  end subroutine DatamodeUnifiedCalcDriver


  !=============================================================================
  ! Modular Calculation Subroutines
  !=============================================================================

  subroutine calc_convert_precip_accum_to_rate(exportState, rc)
    type(ESMF_State), intent(inout) :: exportState
    integer,          intent(out)   :: rc
    character(len=*), parameter :: subName = 'calc_convert_precip_accum_to_rate: '
    real(r8), pointer :: Faxa_prec(:) => null()

    rc = ESMF_SUCCESS
    call dshr_state_getfldptr(exportState, 'Faxa_prec', fldptr1=Faxa_prec, rc=rc)
    if (ChkErr(rc,__LINE__,u_FILE_u)) return
    
    if (.not. associated(Faxa_prec)) then
      call shr_log_error(trim(subName)//'ERROR: Faxa_prec required.', rc=rc)
      if (ChkErr(rc,__LINE__,u_FILE_u)) return
    end if

    ! Convert from cumulative meters per hour to kg/m^2/s
    ! (rho_water = 1000 kg/m^3, time = 3600 s)
    Faxa_prec(:) = Faxa_prec(:) * (1000.0_r8 / 3600.0_r8)
  end subroutine calc_convert_precip_accum_to_rate


  subroutine calc_convert_rad_accum_to_flux(exportState, rc)
    type(ESMF_State), intent(inout) :: exportState
    integer,          intent(out)   :: rc
    character(len=*), parameter :: subName = 'calc_convert_rad_accum_to_flux: '
    real(r8), pointer :: Faxa_swdn(:) => null()

    rc = ESMF_SUCCESS
    call dshr_state_getfldptr(exportState, 'Faxa_swdn', fldptr1=Faxa_swdn, rc=rc)
    if (ChkErr(rc,__LINE__,u_FILE_u)) return
    
    if (associated(Faxa_swdn)) then
      Faxa_swdn(:) = Faxa_swdn(:) / 3600.0_r8
    end if
  end subroutine calc_convert_rad_accum_to_flux


  subroutine calc_partition_sw_4band(exportState, rc)
    type(ESMF_State), intent(inout) :: exportState
    integer,          intent(out)   :: rc
    character(len=*), parameter :: subName = 'calc_partition_sw_4band: '
    
    real(r8), pointer :: Faxa_swdn(:) => null()
    real(r8), pointer :: Faxa_swvdr(:) => null()
    real(r8), pointer :: Faxa_swvdf(:) => null()
    real(r8), pointer :: Faxa_swndr(:) => null()
    real(r8), pointer :: Faxa_swndf(:) => null()
    
    rc = ESMF_SUCCESS
    ! (Fetch pointers and validate them here...)
  end subroutine calc_partition_sw_4band


  subroutine calc_partition_precip_freezing(exportState, rc)
    type(ESMF_State), intent(inout) :: exportState
    integer,          intent(out)   :: rc
    character(len=*), parameter :: subName = 'calc_partition_precip_freezing: '
    
    real(r8), pointer :: Faxa_prec(:) => null()
    real(r8), pointer :: Faxa_prrn(:) => null()
    real(r8), pointer :: Faxa_prsn(:) => null()

    rc = ESMF_SUCCESS
    call dshr_state_getfldptr(exportState, 'Faxa_prec', fldptr1=Faxa_prec, rc=rc)
    if (ChkErr(rc,__LINE__,u_FILE_u)) return
    if (.not. associated(Faxa_prec)) then
      call shr_log_error(trim(subName)//'ERROR: Faxa_prec required.', rc=rc)
      if (ChkErr(rc,__LINE__,u_FILE_u)) return
    end if

    call dshr_state_getfldptr(exportState, 'Faxa_prrn', fldptr1=Faxa_prrn, rc=rc)
    if (ChkErr(rc,__LINE__,u_FILE_u)) return
    call dshr_state_getfldptr(exportState, 'Faxa_prsn', fldptr1=Faxa_prsn, rc=rc)
    if (ChkErr(rc,__LINE__,u_FILE_u)) return

    ! Check associations and partition...
  end subroutine calc_partition_precip_freezing

end module cdeps_datamode_unified_mod

@NickSzapiro-NOAA

@NickSzapiro-NOAA this is a fantastic idea! Can you confirm that we could apply successive calculation subroutines, for example: convert units for shortwave then split in 4 bands?

Another useful feature would be to be able to calculate new fields from input variables (particularly useful for thermodynamics variable such as the various flavors of humidity,...) It does not have to be done in the immediate future but building a framework that would allow a future implementation would be amazing.

@raphaeldussin

Yes @raphaeldussin

To make it concise+robust, I'm thinking rule is that user would first initialize all export variables they want via variables in input stream (initializing to something else, if what you want isn't present) like:

 # in datm.streams
stream_data_variables01: "temperature Sa_tbot" "specific_humidity Sa_shum" "surface_pressure Sa_pslv" "surface_pressure Sa_pbot" "downward_solar Faxa_swdn" "downward_longwave Faxa_lwdn" "wind_speed Sa_wspd" "precipitation Faxa_rain" "precipitation Faxa_snow" "downward_solar Faxa_swvdr" "downward_solar Faxa_swvdf" "downward_solar Faxa_swndr" "downward_solar Faxa_swndf"

Then calculate what you need within export state, via scheme switch in calc_opts like

calc_opts = ' calc_convert_rad_accum_to_flux calc_partition_sw_4band calc_partition_precip_freezing '

These schemes should use variables that match UFS field dictionary like CMEPS expects.
If some calculation has to follow another, we should probably enforce that in the code.

I imagine it's usually more practical for users to do these calculations offline and just add to input data, but options can be there.
Maybe in UFS could

link in other components' equations (like surface fluxes, for example).
integrate quality control module to enforce min/max bounds on variables

All this lets CDEPS be "pipes" without needing to know what variables mean physically.
Maybe we'll get tripped up by NaNs, masked values,... but I'm hopeful this would offer helpful infrastructure.

@NickSzapiro-NOAA

@NickSzapiro-NOAA I would like to see a speedy:

Merger of this PR from @uturuncoglu.
Resolution of this issue.
Until then, all the brainstorming ideas are great, but we continue to be stuck without a working solution for us to use the UFS-weather-model with ERA5 forcing fields.

@sanAkel

Thanks for priorities @sanAkel

ESCOMP/CDEPS decision is they don't want #396

For ufs-community/ufs-weather-model#3322 , there are not plans to enable UFS data tests on wcoss2

I will start a draft PR for this soon. It is not the only way to do this but should get us unstuck.
Please let me know if you see better way

@sanAkel

For today @sanAkel , please convert ERA5 input data consistent with GEFS datamode

@sanAkel

For today @sanAkel , please convert ERA5 input data consistent with GEFS datamode

👍 thanks @NickSzapiro-NOAA for your understanding.

@raphaeldussin is starting on the proposed path with GEFS datamode using era.

@NickSzapiro-NOAA

Thank you for sharing these thoughts, @NickSzapiro-NOAA . @mvertens and I talked about this. We currently feel that a generic datamode is not the right approach for CESM or NorESM: We have considered trying to do something like this, but have found that, in practice, there is usually specific code needed for each data mode, and feel that it's better to have the implementation for each data mode be more explicit and straightforward. There can also often be a tradeoff between generality vs. robust error-checking, and some of the verbosity and explicitness that now exists in CDEPS was done deliberately to allow for this more robust error-checking. If we had dozens of atmosphere data modes that just differed in which combinations of processing were used - resulting in significant duplication - then we might feel differently, but with the current small number of datamodes that each have their own requirements, we see more benefit to sticking with the current design.

That said, if UFS wants to introduce a UFS-specific datamode that uses this generic approach - whether something like you've laid out here or something building on #396 - then we are happy for you to do so. We recognize that we have some differences in philosophy from you, and we don't want to hold up your progress because of these different views. We just request that you do this in a way that keeps this new approach contained within a UFS-specific file, so that we can approve it quickly once it's ready without needing to worry about possible interactions with CESM/NorESM code. (This is especially because we are close to major releases of CESM and NorESM, so are trying to minimize risk and don't have much time for reviews at the moment.)

I'll also add, in terms of unit conversion: this recent PR from @mvertens may be helpful: #438

@billsacks

@billsacks I am new to all this, but I do catch up fast!

At some stage, it will be good to know

We currently feel that a generic datamode is not the right approach for CESM or NorESM:

why?

but have found that, in practice, there is usually specific code needed for each data mode, and feel that it's better to have the implementation for each data mode be more explicit and straightforward.

Yes, that happens (often)!

We just request that you do this in a way that keeps this new approach contained within a UFS-specific file, so that we can approve it quickly once it's ready without needing to worry about possible interactions with CESM/NorESM code.

Thanks!

At some stage, it will be good to know

We currently feel that a generic datamode is not the right approach for CESM or NorESM:

why?

Broadly speaking, for the reasons I mentioned above.

It's possible that, after the UFS has developed an approach, we'll see some advantages to that approach. But for now - especially as close as we are to major releases - we want to stick with more conservative approaches that follow the current design.

@billsacks

Thanks for guidance @billsacks @mvertens . I do not anticipate that changes will impact your models.

Best wishes for everyone's efforts to be successful

I am concerned that there is a pattern of bias towards your models though ... private conversations deciding won't fix / won't review / don't touch are not appropriate for a community model and branch, imo

@billsacks

Thanks for guidance @billsacks @mvertens . I do not anticipate that changes will impact your models.

Best wishes for everyone's efforts to be successful

I am concerned that there is a pattern of bias towards your models though ... private conversations deciding won't fix / won't review / don't touch are not appropriate for a community model and branch, imo

I 💯agree with you @NickSzapiro-NOAA while I totally get your point @billsacks

As for my issues, I hope to get ourselves unstuck too.

I will sure circle back here on this issue later.

Thanks, everyone. I really appreciate the ideas here, and Nick, I hear the concern about process too. I think there's real value in a follow-up discussion that digs into the benefits and challenges of a more unified data mode, and where it might fit alongside the current design. That's easier to do well in a live conversation than in an issue thread, so I'll email everyone here to set up a meeting for anyone interested, and we can post a summary of the discussion back on this issue so it stays visible to all.

Navigation Menu

Repository navigation

User-configurable datamode for UFS #442

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

User-configurable datamode for UFS #442

Description

Activity

raphaeldussin commented on Sep 16, 2026

NickSzapiro-NOAA commented on Sep 16, 2026

sanAkel commented on Sep 16, 2026

NickSzapiro-NOAA commented on Sep 16, 2026

NickSzapiro-NOAA commented on Sep 16, 2026

sanAkel commented on Sep 16, 2026

billsacks commented on Sep 16, 2026

sanAkel commented on Sep 16, 2026

billsacks commented on Sep 16, 2026

NickSzapiro-NOAA commented on Sep 17, 2026

sanAkel commented on Sep 17, 2026

briandobbins commented on Sep 21, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions