feat: address migration gaps (WIP)

[pjwerneck]

Summary

Closes most of the migration gaps that block safe upgrades: private Drive folders are adopted by rename, P2P folders are found regardless of the version in the name, and durable surfaces (checkpoints, rolling state, crypto keys, peers JSON, local caches) carry or refuse versions instead of failing silently.

Adds a per-protocol minimum-supported floor (shipped inert at "0"), stops refusing peers on client semver mismatch, and hardens release/CI (export → publish → tag → bump, client migration tests in CI, numeric version ordering). Job protocol skew delivery is covered end-to-end; dataset transport multi-copy (A1) and a few process/policy items remain open.

Note

The base branch is stephen/syft-client-migrations, not dev. This branch assumes that branch will be merged first.

Changes

Done

• A1 dataset transport. create_dataset writes one copy for each protocol
  version in the audience. Each copy gets its own Drive collection, and every
  collection reaches the whole audience. A peer that upgrades later therefore
  moves to the newer layout with no action by the owner. The collection name
  holds the version as a v<n> infix before the separator. A client that
  predates this change therefore never lists a layout it cannot read. A
  protocol-0 name does not change. The client uploads the private data of a
  copy with that copy. A watcher keeps the newest readable layout for each
  dataset, and warns and skips the rest.
• A2a private folders. _find_or_adopt_versioned_folder renames the folder
  of the highest earlier version and keeps the data. The personal datasite
  folder, the checkpoints folder, and the rolling-state folder all use it.
• A2b P2P folders. _find_p2p_folder_id finds a folder whatever version
  its name holds. _filter_patch_compatible is gone.
• A3 floor. Each package publishes MIN_SUPPORTED_*_PROTOCOL_VERSION. The
  value travels as min_supported_version, and negotiate_protocol_version
  enforces it for jobs and for datasets.
• A3 peer gate. An incompatible client version no longer skips a peer
  (get_peer_compatibility_status). force_ignore_protocol_version is gone.
• A3 login. The mismatch prompt keeps local and remote data. A full
  deletion is an explicit second choice. A run with no terminal takes the
  keep-data answer, so a notebook or a scheduled run does not stop. Login
  writes the version file on both sides. A second login therefore shows no
  prompt, and a peer reads a current version.
• A4. The dead ClientVersionMismatchError and
  ProtocolVersionMismatchError are gone. The inert semver PROTOCOL_VERSION
  field stays on the wire.
• A5. The defaults for an unknown peer are documented, and a forced path
  logs a warning. See test_unknown_peer_forced_path.py.
• B1. The client refuses a checkpoint or a rolling state from a later
  client, and downloads all events instead. A local load that fails reports the
  failure.
• B2. The crypto key file carries CRYPTO_KEYS_VERSION. A later version is
  refused, because a private key cannot be rebuilt.
• B3. SYFT_peers.json carries a version under _meta. The client logs an
  unknown peer state and skips it, and the rewrite of the raw map keeps the
  entry.
• B4. PersistedDict writes {"version", "entries"}. An old flat file
  reads as version 0, and a later version resets to empty.
• C1. The release job exports the artifacts, publishes, tags, then bumps.
  --dependents published pins the published version, and
  protocol_bump_missing compares against the newest released protocol.
• C2. The client migration tests run in CI, with
  just test-client-migrations, and again after a release.
• C3 cases 1 and 2. A job that is negotiated down to protocol 0 arrives and
  reads (test_job_protocol_skew_delivery.py). A dataset reaches an audience
  of mixed protocol versions (test_dataset_multicopy_delivery.py).
• D1. _version_order compares a version as a number, so "10" is after
  "2".
• D2. Tests guard the additive-only rule for the VersionInfo fields
  (test_version_info_fields.py).
• D3. The job and the dataset negotiation tests assert the identity of the
  live peer-schema dict.

Open items

One decision on A3 clears the first group. One release
cycle clears the second.

1. Waits on a decision (A3)

A peer can speak a protocol below the floor of a package. The client can refuse
it, warn about it, or go read-only. That question is open, and three items wait
on the answer. Every floor is "0" today, so the mechanism refuses nothing and
no user sees a difference.

• A3 policy. The rule itself: refuse, warn, or read-only.
• A3 residual. target_protocol_versions_for_peers takes
  min(ours, theirs) and applies no floor. A1 makes this path live. The
  correct action for a peer below the floor is part of the same decision.
• C3 case 3. An upgrade in place with mixed peers has no automated test,
  because the correct result is not defined without the policy. A manual test
  covers this ground now.

2. Closed by the next release

The first two items block the release. Do them before it. No code change on
this branch can close any of the three, because each one needs a release that
happened.

• syft-migration is not on PyPI, and no CD workflow releases it.
  syft-client, syft-job, and syft-dataset all depend on it with no pin. The
  monorepo resolves the dependency from the workspace, so a local install
  works. An install of the next release fails.
• The one-time version increase. The branch still names the published
  versions: syft-client 0.1.117, syft-job 0.1.39, syft-dataset 0.1.20. The
  release job publishes the version that is on the branch, and PyPI refuses a
  version that exists.
• The 0.1.117 fixture and package artifact. Both hold
  syft_client_install_source and attestation_token. The published wheel does
  not define these fields, because PyPI received it 12 days before the fields
  arrived. A correct file needs the published wheel as the source.

3. Needs a rehearsal by a person

• C4 and E1. Nobody followed the migration guidance end to end, and only
  VersionInfo has a real migration pair. Both items close when one person
  does a full object version increase on a branch that never merges. That
  person records each point where the tools say nothing. A synthetic V2 must not
  merge: a released V2 freezes its schema and owes a migration forever.

4. Open by decision

• The provisional protocol lock (C1). It is a workflow contract for the whole team, not a part of the migration system. The cost of the decision is measured and written down.

5. Permanent, and not work

• The semver PROTOCOL_VERSION field (A4). It carries no meaning, and it
  must stay on the wire, because a supported older client parses it.

6. A rare failure, kept loud on purpose

• A2c. Two compatible private folders reach _expect_one, which raises and
  names both folders. An automatic merge can put the data of a user out of
  reach, so the client stops instead. Folder adoption makes this case rare.

7. Not measured

• B4 residual. A reset of file_hashes replays the event stream. The reset
  works. The cost for a large datasite is not measured.

Testing

• Unit and migration tests for each gap

Asana task

https://app.asana.com/1/1185126988600652/project/1216249688888494/task/1217005273770094?focus=true