Skip to content

Interpreter restart: Py_Finalize is viable, C extensions are the blocker #61

Open

Description

@zcsizmadia

Investigation of the "interpreter restart" roadmap item, probed against real interpreters on Python 3.14.4 and 3.15.0rc1.

Conclusion: the blocker is real, but it is not the one recorded in the code.

What the code currently says

PyRuntime.Shutdown deliberately skips Py_Finalize, and explains why:

On Python 3.13+ the internal stop-the-world GC that runs inside Py_Finalize() calls _PyThreadState_Attach on the current thread. Because we have already attached a thread state via PyEval_RestoreThread (or PyGILState_Ensure), that second attach triggers the fatal error:
Fatal Python error: _PyThreadState_Attach: non-NULL old thread state

That did not reproduce

Four scenarios, both versions, all clean — no fatal error:

scenario 3.14.4 3.15.0rc1
Py_Initialize → Py_Finalize ok ok
thread state attached via PyGILState_Ensure and held, then Py_Finalize ok ok
PyEval_SaveThread → PyEval_RestoreThread → Py_Finalize ok ok
full restart: finalize, re-initialize, import and run again ok ok

Either the upstream defect was fixed, or it needed a narrower condition than a bare probe reproduces.

This is not yet grounds to change Shutdown. The probe is a bare interpreter; the real path also has the asyncio host, the object registry and loaded extensions in play. It only establishes that Py_Finalize is not inherently fatal on current CPython.

The actual blocker

Re-importing a C extension after a restart fails on both versions:

Py_Initialize  →  import numpy (2.4.4)  →  Py_Finalize  →  Py_Initialize
  →  import numpy
     ImportError: cannot load module more than once per process

Single-phase-init extension modules cannot be loaded twice in one process. That is upstream and outside PyDotNet's control.

So restart splits by workload:

  • Pure-Python — viable today
  • Anything importing numpy, pandas, torch, pyarrow — not viable, and no amount of work here changes that

Given PyDotNet exists largely to reach those libraries, a general "restart the interpreter" feature would be a trap: it would appear to work in tests and fail on the first real workload.

Worth noting: same blocker as sub-interpreters

The roadmap's sub-interpreter item is gated on exactly the same constraint — an extension must opt in to multi-phase initialisation to be usable in a sub-interpreter, and the same modules are the problem. One piece of research answers both, and if the ecosystem moves, both unlock together.

Suggested position

  1. Do not implement general restart. The extension constraint makes it misleading rather than useful.
  2. Update the comment in Shutdown to cite the extension problem, which is verified and permanent-ish, rather than the fatal error, which did not reproduce. The current text will send the next reader down the wrong path.
  3. Revisit alongside sub-interpreters, since ecosystem support for multi-phase init decides both.

Reproduction probe is in the session scratchpad; happy to commit it under benchmarks/ or samples/ if it is worth keeping.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions