Repository navigation
Description
Investigation of the "interpreter restart" roadmap item, probed against real interpreters on Python 3.14.4 and 3.15.0rc1.
Conclusion: the blocker is real, but it is not the one recorded in the code.
What the code currently says
PyRuntime.Shutdown deliberately skips Py_Finalize, and explains why:
On Python 3.13+ the internal stop-the-world GC that runs inside
Py_Finalize()calls_PyThreadState_Attachon the current thread. Because we have already attached a thread state viaPyEval_RestoreThread(orPyGILState_Ensure), that second attach triggers the fatal error:
Fatal Python error: _PyThreadState_Attach: non-NULL old thread state
That did not reproduce
Four scenarios, both versions, all clean — no fatal error:
| scenario | 3.14.4 | 3.15.0rc1 |
|---|---|---|
Py_Initialize → Py_Finalize |
ok | ok |
thread state attached via PyGILState_Ensure and held, then Py_Finalize |
ok | ok |
PyEval_SaveThread → PyEval_RestoreThread → Py_Finalize |
ok | ok |
| full restart: finalize, re-initialize, import and run again | ok | ok |
Either the upstream defect was fixed, or it needed a narrower condition than a bare probe reproduces.
This is not yet grounds to change Shutdown. The probe is a bare interpreter; the real path also has the asyncio host, the object registry and loaded extensions in play. It only establishes that Py_Finalize is not inherently fatal on current CPython.
The actual blocker
Re-importing a C extension after a restart fails on both versions:
Py_Initialize → import numpy (2.4.4) → Py_Finalize → Py_Initialize
→ import numpy
ImportError: cannot load module more than once per process
Single-phase-init extension modules cannot be loaded twice in one process. That is upstream and outside PyDotNet's control.
So restart splits by workload:
- Pure-Python — viable today
- Anything importing numpy, pandas, torch, pyarrow — not viable, and no amount of work here changes that
Given PyDotNet exists largely to reach those libraries, a general "restart the interpreter" feature would be a trap: it would appear to work in tests and fail on the first real workload.
Worth noting: same blocker as sub-interpreters
The roadmap's sub-interpreter item is gated on exactly the same constraint — an extension must opt in to multi-phase initialisation to be usable in a sub-interpreter, and the same modules are the problem. One piece of research answers both, and if the ecosystem moves, both unlock together.
Suggested position
- Do not implement general restart. The extension constraint makes it misleading rather than useful.
- Update the comment in
Shutdownto cite the extension problem, which is verified and permanent-ish, rather than the fatal error, which did not reproduce. The current text will send the next reader down the wrong path. - Revisit alongside sub-interpreters, since ecosystem support for multi-phase init decides both.
Reproduction probe is in the session scratchpad; happy to commit it under benchmarks/ or samples/ if it is worth keeping.
Activity