Revalidate legacy tenant-repo checkpoints after fail-closed ingestion upgrade

Context

#15748 / PR #15752 fixed future tenant-repo syncs so an ingestion summary containing errors can no longer advance lastIngestedRev. It also added a safe, explicitly scoped --full --repo-slug <slug> operator path.

That repair is forward-correct but not self-healing for already-persisted checkpoints. Older deployments may have stored a repository head after an error-bearing ingest, then skipped every later poll because Git stayed at that head. Upgrading the code changes future write semantics but leaves the old checkpoint indistinguishable from one produced by an error-free ingest.

Live latest-open sweep: checked the latest 20 open issues at 2026-07-23T15:08:32Z; no equivalent found. The recent all-state A2A claim sweep found no overlapping lane. Semantic and exact-history sweeps found reconciliation/full-reingest predecessors and #15748, but no ticket for checkpoint-contract versioning or automatic legacy revalidation.

The Problem

The current per-repo persisted state contains:

{
    lastIngestedRev,
    lastRunAttemptAt,
    consecutiveFailures
}

It carries no proof of which ingestion-success contract authorized lastIngestedRev. A pre-fix checkpoint and a post-fix checkpoint have the same shape.

Consequently, an upgrade can remain permanently stuck at zero or partial documents when:

1. an old run ingested with returned errors;
2. the old implementation still persisted the repository head;
3. the repository has not changed since;
4. the new scheduler sees lastIngestedRev === HEAD and has no reason to replay.

The only current recovery is an in-container CLI full replay. That assumes shell/server access and operator knowledge of every affected repo, which is not a stable cloud-upgrade contract.

The Architectural Reality

• The existing orchestrator tenant-repo lane owns repository acquisition, envelope construction, error-free-summary validation, backoff, and checkpoint persistence. Recovery belongs there, not in a remote KB/MC actuator.
• fullReplay already defines the safe semantic: build from a null revision base, require explicit repo scope, and preserve the old checkpoint until the replay completes without summary errors.
• A legacy checkpoint is not proof of corruption; it is proof of unknown validity. One bounded revalidation replay is the safe migration cost.
• Revalidating every repo simultaneously would create a deployment-wide CPU/network/embedding burst. Migration must reuse cadence, jitter, concurrency, and backoff to remain gradual.
• Public diagnostics must expose migration state without repository identities or raw errors.

Structure-map ownership: existing TenantRepoSyncService, tenant-repo scheduling helpers, revision persistence, deployment-state bridge, and cloud tenant-ingestion docs. No new daemon, MCP write tool, or service boundary is required.

The Fix

1. Version the persisted checkpoint success contract, for example with a per-repo ingestContractVersion or equivalent proof marker written only after an error-free ingestion summary.
2. Treat entries that lack the current proof marker as replay-required rather than as trusted incremental bases.
3. Let the existing periodic lane perform a bounded full replay for legacy entries from a null revision base. Admit migration work gradually through existing cadence/jitter/backoff/concurrency controls; do not burst all repos on first boot.
4. On replay failure, preserve the previous head, keep the entry replay-required, advance retry/backoff state, and retry later.
5. On replay success, atomically persist the new head plus the current success-contract marker and resume normal incremental sync.
6. Project aggregate and hashed per-repo migration state through deployment diagnostics so an upgrade can prove whether legacy revalidation is pending, failing, or complete.
7. Keep the manual scoped full-replay CLI as an operator override, not the required migration path.

Contract Ledger Matrix

Decision Record impact

Aligned with ADR 0014's orchestrator-owned tenant-repo scheduler. No ADR amendment: this is a persisted-state migration and recovery policy at the existing lane boundary.

Acceptance Criteria

• Per-repo checkpoint state records a versioned proof that its head was authorized by the current error-free ingestion contract.
• Legacy entries without that proof are classified as replay-required; they are not used as trusted incremental bases.
• The periodic lane performs bounded null-base replay for legacy entries without requiring shell access or a remote write tool.
• Migration work is staggered through existing cadence/jitter/backoff/concurrency controls; a deployment with many repos does not replay all of them simultaneously on boot.
• A thrown ingest failure or returned error-bearing summary preserves the old head, leaves migration pending, increments failure/backoff state, and retries later.
• A clean replay atomically persists the new head plus current proof marker and resumes incremental sync.
• Mixed legacy/current repositories remain isolated: current entries continue normal cadence while legacy entries revalidate.
• An old-but-valid checkpoint incurs at most one harmless bounded replay before becoming current.
• Restart/recreate persistence preserves pending migration and failure/backoff state.
• Deployment diagnostics expose pending/failed/complete counts and hashed per-repo state without tenant/repo names, clone URLs, refs, credentials, or raw errors.
• Tests cover a poisoned legacy checkpoint, a valid legacy checkpoint, repeated failure, clean recovery, mixed state, and restart continuity.

Out of Scope

• A public MCP replay/write actuator or remote shell surface.
• Inferring which historical checkpoints were definitely poisoned.
• Deleting or hand-editing the revisions file.
• Replacing the manual scoped full-replay CLI.
• Changing tenant-repo acquisition, parser selection, or embedding models.

Avoided Traps

• Do not trust an unversioned checkpoint merely because its repository head still exists.
• Do not erase the old checkpoint before a clean replay; it remains the last known cursor and recovery evidence.
• Do not replay the entire deployment in one first-boot burst.
• Do not treat “legacy” as “corrupt”; the contract is unknown validity followed by one proof-producing replay.
• Do not smuggle remote mutation authority into KB/MC to solve an orchestrator-owned migration.

Related

• #15748 / PR #15752 — forward fail-closed checkpointing and manual scoped replay.
• #11789, #11790, #11942 — tenant-repo envelope, scheduler, and persisted retry-state lineage.
• #15759 — canonical cloud persistence for the orchestrator state directory containing this checkpoint file.

Origin Session ID: fc1a49c1-e30a-4e3a-960a-e0596367a4c1

Handoff Retrieval Hint: legacy tenant repo checkpoint ingest contract version replay-required automatic full replay fail closed