Repository navigation
Open
Description
Just writing up a bunch of impressions while they’re fresh in my mind. I’ll look through them in the next few days and see which ones are actionable and e.g. deserve their own issues. Comments welcome!
- A few learners were using
line_profiler.pyorpandas.pyas file names for those examples ➔ that leads to errors because the code itself is running e.g.import pandasand Python prioritizes the local file over the installed module- There’s a specialised error message for this since Python 3.13—another reason to always upgrade to the latest version, if you can 😉
- In my slides, I used the FFEA case study as an example when transitioning from function-level to line-level profiling (Roughly: “Sometimes you can tell by looking at the function-level results that there’s an issue—like in this example, where the Rod initialiser function was called millions of times more often than expected, because we were throwing away old rods and creating new ones, instead of updating them. Other times you need to dig deeper and that’s what line-level profiling is for.”) Then, towards the end of the optimisation section in the heat grid example, I could refer back easily (“The first implementation is creating a new grid and throwing the old one away for every step; kind of like the rods we mentioned earlier. If we stick with two grids and update them instead …”), which I think was helpful didactically, because it helps learners see this as a general pattern instead of a one-off example.
- For parallelisation: ➔ moved these to Add lesson on parallelisation #8
- We did discuss a bit about how submitting batch jobs to the cluster is already a type of parallelisation, which is external to Python.
- Someone also asked about parallelisation within the linear algebra libraries underlying NumPy. We have a callout box on that in the NumPy lesson already; might be worth cross-referencing if/when we add a parallelisation episode
- Someone else asked whether it’s safe to parallelise code that’s writing to the same dataframe. The answer is probably just “in general, no; but check the documentation for details on how to handle that”, but that’s something we should include in a parallelisation episode.
- In the pandas/vectorisation exercise:
- We could emphasize more strongly how the solution is basically just taking the existing
pythagorasfunction and replacingdf['f_vertical']withvertical. Maybe switch fromnp.sqrt()to**0.5to make this even more explicit? - Talking about “the vertical column” and “the horizontal column” is slightly awkward. 😅 Time to bikeshed column names? (My first thought was latitude/longitude, but that implies a spherical geometry, so Pythagoras is no longer valid; maybe length/width would work?)
- Maybe add a small visualisation here, just to illustrate the shape of that dataframe; and that “columns are numpy arrays; all these functions are iterating over rows”? I noticed a few learners struggle with visualising what’s going on in that code example; and I remember it took me quite a while to thoroughly understand that when I first looked through the materials, too.
- We could emphasize more strongly how the solution is basically just taking the existing
- Doing a live demo of the parallelised downloads example is scary; I’ve had a brief network glitch the first time I tried it, so I got a bunch of error messages. It worked a few seconds later when I tried a second time; but oof … That probably deserves an instructor note.
Metadata
Metadata
Assignees
Labels
No labels
Activity
Robadob commentedon Feb 26, 2026
Specifically task/job arrays? (possibly useful to stick to SLURM terminology of job array)
Otherwise notes seem reasonable.
Re parallel guidance, by default most advanced data structures are multiple read, single write (but check documentation). Data tables are probably more of a static data-structure, so parallel writes are likely to be possible at a data-structure level, though parallel writes to the same cell won't be race condition free (unless pandas has an atomic write option).
Robadob commentedon Feb 26, 2026
Yeah go for it, the names are kind of irrelevant, could even just be
x,y.JostMigenda commentedon Mar 16, 2026
Two comments from learners from our feedback form:
I think the small toy examples (fizzbuzz, bubblesort, is_prime) are okay—there, the focus is really not on what the code does, but on how to use the profiling tools.
For the predprey model, though, since we’re using that several times throughout the course a bit of explanation would be useful. I’ll add a few sentences on agent-based models and concrete research applications, to provide more context to learners.
Robadob commentedon Mar 16, 2026
PrepPrey is not anything generic computer science, it's more generic agent based modelling (which is a relatively accessible field imo).
The difficulty is that for a proper profiling experience, they need to be profiling some non-trivial code. That was the simplest example I could come up with. Adding an optional extended explanation could work, but it shouldn't be strictly necessary.
JostMigenda commentedon Mar 16, 2026
I agree; but the fact that two learners independently got the wrong impression indicates that we need to make this clearer. Part of that is on me for not explaining that well during the course, but adding a bit more context on research applications of ABMs should help.
This (and a few others, checked off in the original post above) is contained in #42.
JostMigenda commentedon Apr 20, 2026
Maybe those two would belong together in an “Avoid unnecessary work” section? That would also allow us to refer back to the predator/prey example (in
Grass.eaten, where it’s checking once per grass agent whether each prey is hungry—even though ~80% of them aren’t; so it might be more efficient to pre-compute that once per round and just pass in the list of hungry prey toGrass.eaten).Another fairly common issue like that is recalculating values in a for-loop that don’t change between iterations of the loop.
Robadob commentedon Apr 20, 2026
I always refer to it as redundancy (though arguably all optimisations are a case of removing redundant work, merely how indirect the redundancy is). I've found it difficult to really write much on these obvious cases like FFEA returning redundant temporary copies, because I feel it comes off patronising. If someone wrote the code and that's a significant cost, they should catch it straight away with profiling results.
If you feel you can draft something less blunt than me, go for it.