Skip to content

Add page on parallelisation #60

Closed as duplicate
Closed as duplicate

Description

@JostMigenda

In #59 I’ve added a code sample using a concurrent.futures.ThreadPoolExecutor to parallelise downloading files.

I think it would be good to expand on that and have a whole page with more information:

  • Explain the GIL (as of Python 3.13, it can be disabled, but this is not yet ready for general usage: requires a separate Python build, lack of compatibility with modules using the C-API, 5–10% slowdown for single-threaded code)
  • Difference between ThreadPoolExecutor and ProcessPoolExecutor
  • async/await
  • Clear flowchart of which method to use when

I’ve got an example for ThreadPoolExecutor already (in #59); for ProcessPoolExecutor I could adapt an example from an astrophysics code I’ve developed.

I’m not sure about how to handle async/await. Since it’s conceptually more challenging, maybe it’s better to keep that section very short and high level? And just say “if you have a use case for this [enumerate what those would be], then see the following links for further information”?

Activity

  1. Robadob commented on Mar 9, 2025

    @Robadob
    Member

    Yes, whilst the new example from #59 is clearly worth showing it feels like it's missing some context that more novice course attendees would benefit from.

    Similar applies to your astrophysics code here, it would be some of the harder to parse(?) code from the course so some handholding would be preferable (and maybe more detail in an instructor note?).

    I’m not sure about how to handle async/await. Since it’s conceptually more challenging, maybe it’s better to keep that section very short and high level?

    Yeah something to the effect of, "more advanced parallel concepts such as x,y exist in Python however these are outside the scope of the course (would require a course of their own!)...learn more here".

  2. JostMigenda commented on Aug 21, 2025

    @JostMigenda
    CollaboratorAuthor

    Let’s continue discussion in the main repo at carpentries-incubator#8.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions