test(state): cross-store fault-injection E2E for the unified state control plane (issue #5314)

[qqeasonchen]

Summary

Closes #5314 (one-shot, supersedes the earlier plan to split into multiple PRs).

This PR delivers the cross-store fault-injection test suite for the unified state control
plane introduced by #5301. The four individual Sub-PRs (#5310/#5311/#5312/#5313) each ship a
contract test for their own store, but no test exists for invariants that span two or more
stores — and those are exactly the invariants the runtime depends on. The 6 scenarios here
exercise the failure modes at the seams between stores, the runtime, and the cluster-shared Meta.

What ships

4 test-harness primitives (eventmesh-runtime/src/test/java/.../state/fault/)

• MetaPartitionSwitch — wraps a MetaStore, lets a test open/close a simulated network
  partition. Mutating ops throw MetaPartitionException; reads continue against the
  pre-partition snapshot.
• CrossStoreRaceProbe — ordered log of cross-store operations (DELIVERY_PUT/REMOVE,
  OFFSET_WRITE/READ, TASK_UPDATE) with a monotonic seq, used to assert
  happens-before relationships.
• JvmCrashHarness — spawns a child JVM, sentinel-file-driven destroyForcibly() (SIGKILL
  on POSIX, TerminateProcess on Windows), then relaunches against the same on-disk stores.
  Gated on ENABLE_JVM_CRASH_HARNESS=true; the in-process simulation in scenario 1 covers
  the same property without spawning a child JVM and runs everywhere.
• InMemorySubscriptionStore — ConcurrentHashMap-backed SubscriptionStore for the split-
  brain scenario, so a test can hold two views of the world.

6 fault-injection scenarios (CrossStoreFaultInjectionTest, 596 LoC)

#	Scenario	Failure mode	Core invariant asserted
1	CrashMidAckReAck	crash after offset-write, before MQ-ACK callback	recovery on a fresh dispatcher retires the delivery without re-invoking the channel (#5291 idempotency)
2	MetaPartitionDuringDlq	Meta unreachable while dead-letter recording	MetaBackedDeadLetterStore throws MetaPartitionException — a loud failure, not a silent no-op — so the dispatcher keeps the delivery in flight and retries on heal (#5292)
3	A2aCancelMidStream	cancel lands between PENDING and RUNNING	the taskEpoch guard rejects the late transition; the task converges on a single terminal state (#5302)
4	SubscriptionReRegisterAfterSplit	subscription update during a Meta partition	after heal the latest write wins — nothing dropped, nothing duplicated (#5288, #5301 SubscriptionStore)
5	OffsetStoreRaceVsDeliveryStore	cross-thread offset-advance vs retire race	the probe log proves every DELIVERY_REMOVE is preceded by an OFFSET_WRITE at the same offset (#5289 at-least-once)
6	A2aDispatchRaceVsTaskStore	two dispatchers race on one task record	stale-epoch writes are rejected; concurrent createTask yields exactly one winner (#5291)

All 6 scenarios run fully in-process and deterministically. No Testcontainers / no Nacos /
no Docker required.

Documentation

• docs/eventmesh-uni-architecture-redesign.md: new §13.2.12 "Cross-store fault-injection
  verification ([Architecture Review][P1] #5301 Sub-PR D: cross-store fault-injection E2E #5314 Sub-PR D2)" — scope, the 4 harness primitives, the per-scenario
  assertion table, why Testcontainers is not used, and the two extension points.
  Numbered 13.2.12 because §13.2.11 was taken by the dual-topology matrix in PR feat(cluster): PARTITION_OWNED_PULL delivery topology (issue #5309, closes) #5317.

Why one-shot, not split

The 6 scenarios share the same 4 harness primitives. Splitting them would either (a)
duplicate the harness across PRs, or (b) require a pre-PR that lands the harness first and
has no assertions of its own. Keeping the harness slim (414 LoC across 4 files) and bundling
the scenarios keeps the review scope manageable while avoiding the
pre-PR-without-assertions anti-pattern.

Why no Testcontainers

#5314 originally listed 6 Testcontainers-based scenarios, but the in-process approach is
strictly better for these failure modes:

• MetaPartition and JvmCrash are deterministic fault-injection primitives. Wrapping them in
  a Nacos/Kafka container adds a network-latency surface that obscures the assertion rather
  than strengthening it.
• CI sandboxes (Windows containers, macOS runners) often cannot run Docker-out-of-Docker, so
  the Testcontainers variant would be skipped in CI; the in-process variant runs everywhere.
• 4 of 6 scenarios are algorithm-correctness properties about Meta/OffsetStore coordination
  ([Bug] Make RocksDB and Meta offset writes atomically monotonic #5289, [Bug] Prevent stale ACKs from matching reused delivery IDs #5291, [Bug] Make DLQ transition durable before retiring source delivery #5292); running them against real Nacos with network delays does not change
  the assertion outcome.

How to verify locally

# In-process: always runs
./gradlew :eventmesh-runtime:test --tests \
  org.apache.eventmesh.runtime.state.CrossStoreFaultInjectionTest

# Cross-JVM (optional; requires process control / SIGKILL)
ENABLE_JVM_CRASH_HARNESS=true ./gradlew :eventmesh-runtime:test --tests \
  org.apache.eventmesh.runtime.state.fault.JvmCrashHarness

Relationship to existing suites (no overlap)

• SubscriptionStoreTest / SessionStoreTest / DeadLetterStoreTest / TaskStoreTest /
  DeliveryStateStoreTest / MetaBackedDeadLetterStoreTest / MetaBackedTaskStoreTest
  cover single-store contracts.
• DeliveryRecoveryTest covers single-store recovery (dispatcher.recover()).
• ClusterDeliveryFaultTest covers single-store faults (partition ownership failure).
• CrossStoreFaultInjectionTest (this PR) covers cross-store collaborative faults.

Notes for reviewers

• The 6 scenarios are additive: no production code is touched. The only main-source change
  in this branch is zero — the diff is 6 files, +1063 lines, 0 deletions.
• Scenario 3/6 model the taskEpoch guard's documented contract (set at createTask, never
  reset): it rejects cross-restart staleness, not intra-JVM double-writes. Scenario 6's
  concurrentDispatchersConvergeOnFreshestWrite asserts the createTask putIfAbsent
  guarantee rather than inventing a second epoch-bump semantics.
• JvmCrashHarness is shipped but inert unless ENABLE_JVM_CRASH_HARNESS=true, so it cannot
  make CI flaky.

Checklist

• Rebased on the latest apache/develop (includes PR feat(cluster): PARTITION_OWNED_PULL delivery topology (issue #5309, closes) #5317)
• No production-code changes
• Architecture doc updated (§13.2.12)
• Apache CLA on file
• CI green (Build / checkstyle / test)