Skip to content

[Tech Debt] Add end-to-end checkpoint-recovery test for built-in tool-context #836

Closed

Description

@weiqingy

Description

PR #828 made the built-in chat-model tool-call context checkpoint-safe by normalizing non-primitive Python values (UUID, OutputSchema, List[ChatMessage]) to a primitive-only form before they reach sensory memory, and reconstructing the rich types on read. Without this, Pemja wraps those objects as PyObject holders whose JNI pointers go stale after a TaskManager/Python restart, so restoring the checkpointed context SIGSEGVs in JcpPyObject_FromJObject.

The unit tests in #828 assert that the stored form is recursively primitive as a checkpoint-safety proxy, because the bug cannot be reproduced in local/MiniCluster mode: that path never crosses Pemja, and triggering an in-place recovery by throwing an exception does not recreate the JVM, so the failing code path is never exercised.

After #708 lands, we should add an end-to-end test that triggers a real recovery in a standalone cluster — e.g. by killing the TaskManager process so the JVM (and the embedded Python interpreter) is recreated — and verify that the built-in tool-context flow recovers correctly. This would give true before/after verification of the #828 fix rather than relying on the primitive-form proxy.

Depends on #708.

Activity

  1. added
    tech debt[Issue Type] User-unaware issues, such as code refactor and infrastructure maintenance.
    on Jun 11, 2026
  2. weiqingy commented on Jun 14, 2026

    @weiqingy
    CollaboratorAuthor

    When this end-to-end checkpoint-recovery harness lands, it would be a good home for a permanent bytes round-trip assertion: memory.set(k, b"...") in Python → checkpoint → TaskManager restart → memory.get(k) equals the original. #846 admits exact Python bytes into the memory value contract (verified via Pemja source + a local embedded-interpreter probe that materializes bytes as a native Java byte[]), but a true restart round-trip can't run on the MiniCluster, so an e2e assertion here would close the loop.

  3. changed the title [-][Tech Debt] Add end-to-end checkpoint-recovery test for built-in tool-context (true verification of #828)[/-] [+][Tech Debt] Add end-to-end checkpoint-recovery test for built-in tool-context[/+] on Jun 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

fixVersion/0.4.0tech debt[Issue Type] User-unaware issues, such as code refactor and infrastructure maintenance.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions