Skip to content

Add lesson on parallelisation #8

Open

Description

@JostMigenda

This could give a brief overview over ProcessPoolExecutor/InterpreterPoolExecutor /ThreadPoolExecutor, the GIL and free-threading, async/await & a flowchart of which to use when.

See RSE-Sheffield#60 for previous discussion.

Activity

  1. JostMigenda commented on Feb 21, 2026

    @JostMigenda
    CollaboratorAuthor

    See also RSE-Sheffield#72 for previous discussion on Amdahl’s law (which could fit into the parallelisation section), and a draft of a callout box.

  2. JostMigenda commented on Mar 16, 2026

    @JostMigenda
    CollaboratorAuthor

    A few thoughts from #40:

    • mention parallelisation external to Python (e.g., using slurm to submit job arrays to an HPC cluster)
    • Linear algebra libraries underlying NumPy (BLAS/LAPACK) may be parallelised. We have a callout box on that in the NumPy lesson already; might be worth cross-referencing in the parallelisation episode
    • Talk about thread safety (e.g., is it safe to parallelise code that’s writing to the same Python object or data frame?). The answer is probably just “as a first rule of thumb, reading is usually fine and writing isn’t”, but we can refer to official documentation or other resources for details. (As part of the free-threading work, I remember seeing a couple of new resources on that recently.)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions