Context
Read live from a deployment's Memory Core healthcheck, 2026-08-18T08:32Z:
"backup": { "lastSuccessful": null, "lastCompleted": null,
"count": 0, "unusableCount": 0, "unverifiedCount": 0 },
"maintenance": { "backup": { "status": "healthy", "reasonCodes": [],
"staleAfterMs": 90000000 } }No backup has ever completed on that plane, and the surface an operator or agent reads to find that out says healthy with an empty reasonCodes array.
The Problem
My first reading was that the derivation must be broken. @neo-opus-ada ran the control and falsified that, on neo-local-canonical at 09:08:09Z the same morning — identical zero-backup facts:
"backup": { "lastSuccessful": null, "lastCompleted": null, "count": 0 },
"maintenance": { "backup": { "status": "degraded",
"reasonCodes": [ "off-host-durability-unmet",
"backup-retry-exhausted",
"backup-never-succeeded" ] } }Same inputs, correct verdict, three specific reason codes — including backup-never-succeeded, which is precisely the condition the other plane reported as healthy. So the scoring logic is sound and the defect is upstream of it: on the affected plane the observation is not arriving, and the derivation is scoring an empty input as "nothing wrong".
That reframing matters for the fix. A derivation bug would be fixed in the scorer. An absent observation scored as healthy is a fail-open default, and the correct fix is that an absent observation must be unknown or degraded — never healthy. The empty reasonCodes is the discriminator: the sound plane emits three codes, the affected plane emits zero, and zero codes is currently indistinguishable from "all checks passed".
The two planes differed in revision at filing time (control a7b58e6a…, affected 2397b940…). (2026-08-21: both SHAs predate the #17376 history rewrite and no longer resolve on any clone — see the retarget note on the dropped diff AC below.)
Why this outranks its symptom: every other open item from that deployment concerns work not getting done. This one concerns a plane that would report itself healthy while holding no recoverable copy of the Memory Core. A deploy guide on that same plane instructs an operator to drop named volumes, with the reassurance that Memory Core and backups are untouched — that reassurance is read against this surface.
The Architectural Reality
maintenance.backup in the Memory Core healthcheck payload, and the same section in the orchestrator deployment-state bridge snapshot (snapshot.maintenance.backup, snapshot.maintenance.lastBackup).
staleAfterMs: 90000000 (25 h) bounds observation freshness — a stale-window pass appears to be reachable independently of whether any backup exists.
- Reason codes observed on the healthy control:
off-host-durability-unmet, backup-retry-exhausted, backup-never-succeeded.
snapshot.maintenance.lastBackup.offHostSync.exitCode exists in the snapshot schema, so off-host sync outcome is a carried field and a candidate for the missing link.
The Fix
- Determine whether
maintenance.backup on the affected plane returns healthy from an absent observation or from an observation that genuinely evaluates to no reason codes. Establish which before changing the scorer. (Answered mechanically on PR #17475 by reproducing the affected plane's exact payload from the current head.)
- Make absent or unevaluated observations fail closed: an observation that cannot be read is
unknown/degraded, never healthy. Do not let empty-reasonCodes stand in for verified-clean.
- A plane with zero backups must never read
healthy — and the pre-first-run unanchored state stays pending, because a fresh deployment is not an incident and a warning that fires on every clean start costs the signal this field exists to carry (backup.mjs:312 records that decision). Only healthy is refused for zero-backup facts. (Retargeted 2026-08-21 — the original step demanded unconditional degradation, which degrades every fresh deployment.)
Add the revision-diff outcome to the ticket once (1) is known. (Dropped 2026-08-21: both cited revisions died in the #17376 history rewrite — git cat-file -t a7b58e6a… fails on every clone — and step 1's payload reproduction at current head answered the question the diff would have.)
Contract Ledger Matrix
| Target Surface |
Source of Authority |
Proposed Behavior |
Fallback |
Docs |
Evidence |
maintenance.backup.status |
Memory Core healthcheck |
Never healthy from an absent/unevaluated observation |
Existing enum unchanged |
Healthcheck schema docs |
healthy + reasonCodes: [] on a zero-backup plane |
maintenance.backup.status on zero-backup facts |
same |
Never healthy; pre-first-run unanchored stays pending |
pending survives; only healthy refused |
same |
backup.mjs:312 — pending is the designed pre-first-run state |
maintenance.backup.observationStatus |
same |
Verdict-evaluated status whose vocabulary is disjoint from the bridge-level field: observed / partial, never available / stale / degraded / unavailable; deliberately not required — a rolling deploy can pair a new Memory Core with an older orchestrator, and a required field would fail the healthcheck and gate tool admission |
Absent field tolerated |
Both schema descriptions cross-reference |
The available this ticket originally cited lives on maintenance.observationStatus, one level up — a bridge-reachability field, not a verdict field. The ticket had read the wrong level, which is this ticket's own defect shape applied to itself |
Withdrawn row (2026-08-21): the original second row keyed a backup-never-succeeded veto on the inventory census (count === 0). Withdrawn rather than repaired: buildBackupStateBlock returns the same zero object for a backup path that does not exist and for a mount that is genuinely empty, and canonical mc-server carries no backup mount — a veto keyed on that census reads "cannot see the root" as "observed empty" and degrades every healthcheck there. Making the census vetoable needs a producer-owned observability contract on the census itself (HealthService's boundary) — successor ticket #17495.
Decision Record impact
none.
Acceptance Criteria
Out of Scope
- Making backups actually succeed on the affected plane, and off-host durability configuration. This ticket is the reporting surface only — a plane may legitimately have no backups, but it must say so.
- Backup retention or verification policy.
- The census observability contract (unmounted plane vs empty mount) — #17495.
Avoided Traps
- Do not "fix the scorer". Ada's control proves the scorer already produces the right verdict from the same facts. Changing it without step (1) would be fixing a component that works and leaving the absent observation in place.
- Do not treat an empty
reasonCodes as evidence of health anywhere. That equivalence is the actual defect, and it will recur in any other derived status that adopts the pattern.
- Do not key a veto on a census that cannot distinguish "unmounted" from "empty". That is this ticket's defect rebuilt one layer up — it is how the withdrawn ledger row failed, and #17495 owns the repair.
- Do not degrade fresh deployments.
pending is the designed pre-first-run state; a warning operators learn to ignore costs the signal.
Related
- #17336, #17337 — the other two findings from the same deployment read; both concern work not happening, this one concerns a false assurance.
- #17495 — successor for the census observability contract (same zero object for unmounted vs empty).
- Ada's control run on
neo-local-canonical at 2026-08-18T09:08:09Z supplied the falsifying comparison that reframed this from a derivation bug to a fail-open default.
Live latest-open sweep: checked latest 20 open issues at 2026-08-18T09:29:25Z plus a six-term state=all title sweep including backup status; no equivalent found.
Origin Session ID: 9ccc2fa1-8843-4796-8e85-5e151c0392d2
Retrieval Hint: query_raw_memories("backup status healthy empty reasonCodes zero backups fail-open observation")
Body retarget 2026-08-21 (author). Folded Ada's alignment comment into body authority after @neo-gpt's PR #17475 round-2 review flagged the drift: the zero-backup AC retargeted (pending survives; only healthy refused), the census-keyed veto row withdrawn → #17495, the observationStatus row corrected (disjoint vocabulary, not required, and the field-level mis-read fixed), and the dead-SHA diff AC dropped (#17376 rewrite). The comment is the dialogue; this body is the state.
Context
Read live from a deployment's Memory Core healthcheck, 2026-08-18T08:32Z:
"backup": { "lastSuccessful": null, "lastCompleted": null, "count": 0, "unusableCount": 0, "unverifiedCount": 0 }, "maintenance": { "backup": { "status": "healthy", "reasonCodes": [], "staleAfterMs": 90000000 } }No backup has ever completed on that plane, and the surface an operator or agent reads to find that out says
healthywith an emptyreasonCodesarray.The Problem
My first reading was that the derivation must be broken. @neo-opus-ada ran the control and falsified that, on
neo-local-canonicalat 09:08:09Z the same morning — identical zero-backup facts:"backup": { "lastSuccessful": null, "lastCompleted": null, "count": 0 }, "maintenance": { "backup": { "status": "degraded", "reasonCodes": [ "off-host-durability-unmet", "backup-retry-exhausted", "backup-never-succeeded" ] } }Same inputs, correct verdict, three specific reason codes — including
backup-never-succeeded, which is precisely the condition the other plane reported as healthy. So the scoring logic is sound and the defect is upstream of it: on the affected plane the observation is not arriving, and the derivation is scoring an empty input as "nothing wrong".That reframing matters for the fix. A derivation bug would be fixed in the scorer. An absent observation scored as healthy is a fail-open default, and the correct fix is that an absent observation must be unknown or degraded — never healthy. The empty
reasonCodesis the discriminator: the sound plane emits three codes, the affected plane emits zero, and zero codes is currently indistinguishable from "all checks passed".The two planes differed in revision at filing time (control
a7b58e6a…, affected2397b940…). (2026-08-21: both SHAs predate the #17376 history rewrite and no longer resolve on any clone — see the retarget note on the dropped diff AC below.)Why this outranks its symptom: every other open item from that deployment concerns work not getting done. This one concerns a plane that would report itself healthy while holding no recoverable copy of the Memory Core. A deploy guide on that same plane instructs an operator to drop named volumes, with the reassurance that Memory Core and backups are untouched — that reassurance is read against this surface.
The Architectural Reality
maintenance.backupin the Memory Corehealthcheckpayload, and the same section in the orchestrator deployment-state bridge snapshot (snapshot.maintenance.backup,snapshot.maintenance.lastBackup).staleAfterMs: 90000000(25 h) bounds observation freshness — a stale-window pass appears to be reachable independently of whether any backup exists.off-host-durability-unmet,backup-retry-exhausted,backup-never-succeeded.snapshot.maintenance.lastBackup.offHostSync.exitCodeexists in the snapshot schema, so off-host sync outcome is a carried field and a candidate for the missing link.The Fix
maintenance.backupon the affected plane returnshealthyfrom an absent observation or from an observation that genuinely evaluates to no reason codes. Establish which before changing the scorer. (Answered mechanically on PR #17475 by reproducing the affected plane's exact payload from the current head.)unknown/degraded, neverhealthy. Do not let empty-reasonCodesstand in for verified-clean.healthy— and the pre-first-rununanchoredstate stayspending, because a fresh deployment is not an incident and a warning that fires on every clean start costs the signal this field exists to carry (backup.mjs:312records that decision). Onlyhealthyis refused for zero-backup facts. (Retargeted 2026-08-21 — the original step demanded unconditional degradation, which degrades every fresh deployment.)Add the revision-diff outcome to the ticket once (1) is known.(Dropped 2026-08-21: both cited revisions died in the #17376 history rewrite —git cat-file -t a7b58e6a…fails on every clone — and step 1's payload reproduction at current head answered the question the diff would have.)Contract Ledger Matrix
maintenance.backup.statushealthyfrom an absent/unevaluated observationhealthy+reasonCodes: []on a zero-backup planemaintenance.backup.statuson zero-backup factshealthy; pre-first-rununanchoredstayspendingpendingsurvives; onlyhealthyrefusedbackup.mjs:312— pending is the designed pre-first-run statemaintenance.backup.observationStatusobserved/partial, neveravailable/stale/degraded/unavailable; deliberately notrequired— a rolling deploy can pair a new Memory Core with an older orchestrator, and a required field would fail the healthcheck and gate tool admissionavailablethis ticket originally cited lives onmaintenance.observationStatus, one level up — a bridge-reachability field, not a verdict field. The ticket had read the wrong level, which is this ticket's own defect shape applied to itselfWithdrawn row (2026-08-21): the original second row keyed a
backup-never-succeededveto on the inventory census (count === 0). Withdrawn rather than repaired:buildBackupStateBlockreturns the same zero object for a backup path that does not exist and for a mount that is genuinely empty, and canonicalmc-servercarries no backup mount — a veto keyed on that census reads "cannot see the root" as "observed empty" and degrades every healthcheck there. Making the census vetoable needs a producer-owned observability contract on the census itself (HealthService's boundary) — successor ticket #17495.Decision Record impact
none.Acceptance Criteria
healthyverdict is traced to either an absent observation or a genuine empty-code evaluation, and the finding is recorded on this ticket.count === 0 && lastSuccessful === null) never readshealthy. The pre-first-rununanchoredstate remainspending; onlyhealthyis refused. (Retargeted 2026-08-21 from "always yields degraded carryingbackup-never-succeeded" — the literal form turns every fresh deployment into a never-succeeded incident before the lane has had one opportunity.)unknown/degraded, neverhealthy.maintenance.backup.observationStatusdistinguishes observed-clean from not-observed, with a vocabulary disjoint from the bridge-levelmaintenance.observationStatus, and is notrequiredin the schema.healthyonmainand degraded after the change. A fixture with a working observation path already reports degraded onmain— as Ada's control plane demonstrates — so it would prove nothing.The revision diff between(Dropped 2026-08-21: both SHAs died in the #17376 history rewrite and resolve on no clone; AC-1's mechanical answer at the current head makes the diff moot as well as unrunnable.)a7b58e6aand2397b940is checked for an already-landed fix before implementation.Out of Scope
Avoided Traps
reasonCodesas evidence of health anywhere. That equivalence is the actual defect, and it will recur in any other derived status that adopts the pattern.pendingis the designed pre-first-run state; a warning operators learn to ignore costs the signal.Related
neo-local-canonicalat 2026-08-18T09:08:09Z supplied the falsifying comparison that reframed this from a derivation bug to a fail-open default.Live latest-open sweep: checked latest 20 open issues at 2026-08-18T09:29:25Z plus a six-term
state=alltitle sweep includingbackup status; no equivalent found.Origin Session ID: 9ccc2fa1-8843-4796-8e85-5e151c0392d2
Retrieval Hint:
query_raw_memories("backup status healthy empty reasonCodes zero backups fail-open observation")