Skip to content

Notes from 2026-02-25 run at KCL #40

Open

Description

@JostMigenda

Just writing up a bunch of impressions while they’re fresh in my mind. I’ll look through them in the next few days and see which ones are actionable and e.g. deserve their own issues. Comments welcome!

  • A few learners were using line_profiler.py or pandas.py as file names for those examples ➔ that leads to errors because the code itself is running e.g. import pandas and Python prioritizes the local file over the installed module
    • There’s a specialised error message for this since Python 3.13—another reason to always upgrade to the latest version, if you can 😉
  • In my slides, I used the FFEA case study as an example when transitioning from function-level to line-level profiling (Roughly: “Sometimes you can tell by looking at the function-level results that there’s an issue—like in this example, where the Rod initialiser function was called millions of times more often than expected, because we were throwing away old rods and creating new ones, instead of updating them. Other times you need to dig deeper and that’s what line-level profiling is for.”) Then, towards the end of the optimisation section in the heat grid example, I could refer back easily (“The first implementation is creating a new grid and throwing the old one away for every step; kind of like the rods we mentioned earlier. If we stick with two grids and update them instead …”), which I think was helpful didactically, because it helps learners see this as a general pattern instead of a one-off example.
  • For parallelisation: ➔ moved these to Add lesson on parallelisation #8
    • We did discuss a bit about how submitting batch jobs to the cluster is already a type of parallelisation, which is external to Python.
    • Someone also asked about parallelisation within the linear algebra libraries underlying NumPy. We have a callout box on that in the NumPy lesson already; might be worth cross-referencing if/when we add a parallelisation episode
    • Someone else asked whether it’s safe to parallelise code that’s writing to the same dataframe. The answer is probably just “in general, no; but check the documentation for details on how to handle that”, but that’s something we should include in a parallelisation episode.
  • In the pandas/vectorisation exercise:
    • We could emphasize more strongly how the solution is basically just taking the existing pythagoras function and replacing df['f_vertical'] with vertical. Maybe switch from np.sqrt() to **0.5 to make this even more explicit?
    • Talking about “the vertical column” and “the horizontal column” is slightly awkward. 😅 Time to bikeshed column names? (My first thought was latitude/longitude, but that implies a spherical geometry, so Pythagoras is no longer valid; maybe length/width would work?)
    • Maybe add a small visualisation here, just to illustrate the shape of that dataframe; and that “columns are numpy arrays; all these functions are iterating over rows”? I noticed a few learners struggle with visualising what’s going on in that code example; and I remember it took me quite a while to thoroughly understand that when I first looked through the materials, too.
  • Doing a live demo of the parallelised downloads example is scary; I’ve had a brief network glitch the first time I tried it, so I got a bunch of error messages. It worked a few seconds later when I tried a second time; but oof … That probably deserves an instructor note.

Activity

  1. Robadob commented on Feb 26, 2026

    @Robadob
    Collaborator

    We did discuss a bit about how submitting batch jobs to the cluster is already a type of parallelisation, which is external to Python.

    Specifically task/job arrays? (possibly useful to stick to SLURM terminology of job array)

    Otherwise notes seem reasonable.

    Re parallel guidance, by default most advanced data structures are multiple read, single write (but check documentation). Data tables are probably more of a static data-structure, so parallel writes are likely to be possible at a data-structure level, though parallel writes to the same cell won't be race condition free (unless pandas has an atomic write option).

  2. Robadob commented on Feb 26, 2026

    @Robadob
    Collaborator

    Talking about “the vertical column” and “the horizontal column” is slightly awkward. 😅 Time to bikeshed column names? (My first thought was latitude/longitude, but that implies a spherical geometry, so Pythagoras is no longer valid; maybe length/width would work?)

    Yeah go for it, the names are kind of irrelevant, could even just be x, y.

  3. JostMigenda commented on Mar 16, 2026

    @JostMigenda
    CollaboratorAuthor

    Two comments from learners from our feedback form:

    The example functions that are typically used in computer science courses are not very useful. For example, the predprey one. I think it's essential to be able to relate to the code to a certain extent

    Some of the worked examples are probably well-known for people with computer science background, but not for me. A little more description on the problem would help understanding.

    I think the small toy examples (fizzbuzz, bubblesort, is_prime) are okay—there, the focus is really not on what the code does, but on how to use the profiling tools.

    For the predprey model, though, since we’re using that several times throughout the course a bit of explanation would be useful. I’ll add a few sentences on agent-based models and concrete research applications, to provide more context to learners.

  4. Robadob commented on Mar 16, 2026

    @Robadob
    Collaborator

    PrepPrey is not anything generic computer science, it's more generic agent based modelling (which is a relatively accessible field imo).

    The difficulty is that for a proper profiling experience, they need to be profiling some non-trivial code. That was the simplest example I could come up with. Adding an optional extended explanation could work, but it shouldn't be strictly necessary.

  5. JostMigenda commented on Mar 16, 2026

    @JostMigenda
    CollaboratorAuthor

    PrepPrey is not anything generic computer science, it's more generic agent based modelling (which is a relatively accessible field imo).

    I agree; but the fact that two learners independently got the wrong impression indicates that we need to make this clearer. Part of that is on me for not explaining that well during the course, but adding a bit more context on research applications of ABMs should help.

    This (and a few others, checked off in the original post above) is contained in #42.

  6. JostMigenda commented on Apr 20, 2026

    @JostMigenda
    CollaboratorAuthor
    • In my slides, I used the FFEA case study as an example when transitioning from function-level to line-level profiling (Roughly: “Sometimes you can tell by looking at the function-level results that there’s an issue—like in this example, where the Rod initialiser function was called millions of times more often than expected, because we were throwing away old rods and creating new ones, instead of updating them. Other times you need to dig deeper and that’s what line-level profiling is for.”) Then, towards the end of the optimisation section in the heat grid example, I could refer back easily (“The first implementation is creating a new grid and throwing the old one away for every step; kind of like the rods we mentioned earlier. If we stick with two grids and update them instead …”), which I think was helpful didactically, because it helps learners see this as a general pattern instead of a one-off example.

    Maybe those two would belong together in an “Avoid unnecessary work” section? That would also allow us to refer back to the predator/prey example (in Grass.eaten, where it’s checking once per grass agent whether each prey is hungry—even though ~80% of them aren’t; so it might be more efficient to pre-compute that once per round and just pass in the list of hungry prey to Grass.eaten).

    Another fairly common issue like that is recalculating values in a for-loop that don’t change between iterations of the loop.

  7. Robadob commented on Apr 20, 2026

    @Robadob
    Collaborator

    Maybe those two would belong together in an “Avoid unnecessary work” section? That would also allow us to refer back to the predator/prey example (in Grass.eaten, where it’s checking once per grass agent whether each prey is hungry—even though ~80% of them aren’t; so it might be more efficient to pre-compute that once per round and just pass in the list of hungry prey to Grass.eaten).

    Another fairly common issue like that is recalculating values in a for-loop that don’t change between iterations of the loop.

    I always refer to it as redundancy (though arguably all optimisations are a case of removing redundant work, merely how indirect the redundancy is). I've found it difficult to really write much on these obvious cases like FFEA returning redundant temporary copies, because I feel it comes off patronising. If someone wrote the code and that's a significant cost, they should catch it straight away with profiling results.

    If you feel you can draft something less blunt than me, go for it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions