Skip to content

feat: enable template histogram creation and collection to accept list of tasks - #421

Open
alexander-held wants to merge 2 commits into
masterfrom
feat/histogram-production-refactor
Open

alexander-held wants to merge 2 commits into
masterfrom
feat/histogram-production-refactor

Conversation

@alexander-held

@alexander-held alexander-held commented Jul 6, 2023 •

Copy link
Copy Markdown
Member

This updates the templates.build, templates.collect, and templates.postprocess implementations to accept a list of templates to produce via a new template_list keyword argument. The full list can be obtained via route.required_templates. By default, all templates will still be processed.

This allows users to parallelize the template histogram production / collection, partially addressing #401. An example of this would be the following:

from dask.distributed import Client, LocalCluster, wait

def produce_single_template(template):
    cabinetry.templates.build(config, template_list=[template])

template_list = cabinetry.route.required_templates(config)

with LocalCluster(n_workers=2) as cluster:
    client = Client(cluster)
    wait(client.map(produce_single_template, template_list))

compared to the non-parallelized version, which is unchanged:

cabinetry.templates.build(config)

breaking change:

  • renamed route.apply_to_all_templates to route.apply_to_templates and changed signature: now takes a list of tasks, which can be generated by route.required_templates
* templates.build, templates.collect, and templates.postprocess now accept a list of tasks
* breaking change: route. apply_to_all_templates renamed to route.apply_to_templates
* breaking change: route.apply_to_templates now requires a list of tasks as argument

Comment thread src/cabinetry/route.py
Comment on lines +43 to +45
TemplateHistogramInformation = Tuple[
Dict[str, Any], Dict[str, Any], Dict[str, Any], Optional[Literal["Up", "Down"]]
]

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This probably would be good to replace by a lightweight class to wrap the information.

@alexander-held
alexander-held force-pushed the feat/histogram-production-refactor branch from 1d83bf7 to fd93dbb Compare September 19, 2023 18:00
@alexander-held
alexander-held force-pushed the feat/histogram-production-refactor branch from 1d29183 to 2f75c8a Compare October 5, 2023 02:47
@alexander-held
alexander-held force-pushed the feat/histogram-production-refactor branch from 2f75c8a to 5536dd8 Compare November 24, 2023 15:54
@alexander-held

Copy link
Copy Markdown
Member Author

This will require a different approach to handle remote workers without access to a shared filesystem. To handle this, histograms would have to be passed back and saved at the head node. The current API does not surface the call to utils._name_and_save, which is quite deep inside internal APIs.

Currently route.apply_to_templates does generic function execution. This might have to be changed to support handling return values and passing those back to the higher level, e.g. templates.build for saving. The downside of this is that saving would only happen (in the non-parallel version) after all templates are constructed (and passed back as histograms).

to-do / think about:

  • is writing at the very end maybe be a problem if some late templates crash
  • memory (or other) issues from keeping all histograms in memory?
  • writing only once at the very end means all histograms can easily be written into a single file (this is a bigger refactor)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant