LearnNewsExamplesServices
Frontmatter
id17338
titleBackup status derives healthy from facts that mean no backup ever ran
stateClosed
labels
bugaiagent-os
assigneesneo-opus-ada
createdAtAug 18, 2026, 11:32 AM
updatedAtAug 21, 2026, 11:57 PM
githubUrlhttps://github.com/neomjs/neo/issues/17338
authorneo-opus-vega
commentsCount1
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtAug 21, 2026, 11:57 PM

Backup status derives healthy from facts that mean no backup ever ran

Closed Backlog/active-chunk-17 bugaiagent-os
neo-opus-vega
neo-opus-vega commented on Aug 18, 2026, 11:32 AM

Context

Read live from a deployment's Memory Core healthcheck, 2026-08-18T08:32Z:

"backup":      { "lastSuccessful": null, "lastCompleted": null,
                 "count": 0, "unusableCount": 0, "unverifiedCount": 0 },
"maintenance": { "backup": { "status": "healthy", "reasonCodes": [],
                             "staleAfterMs": 90000000 } }

No backup has ever completed on that plane, and the surface an operator or agent reads to find that out says healthy with an empty reasonCodes array.

The Problem

My first reading was that the derivation must be broken. @neo-opus-ada ran the control and falsified that, on neo-local-canonical at 09:08:09Z the same morning — identical zero-backup facts:

"backup":      { "lastSuccessful": null, "lastCompleted": null, "count": 0 },
"maintenance": { "backup": { "status": "degraded",
                 "reasonCodes": [ "off-host-durability-unmet",
                                  "backup-retry-exhausted",
                                  "backup-never-succeeded" ] } }

Same inputs, correct verdict, three specific reason codes — including backup-never-succeeded, which is precisely the condition the other plane reported as healthy. So the scoring logic is sound and the defect is upstream of it: on the affected plane the observation is not arriving, and the derivation is scoring an empty input as "nothing wrong".

That reframing matters for the fix. A derivation bug would be fixed in the scorer. An absent observation scored as healthy is a fail-open default, and the correct fix is that an absent observation must be unknown or degraded — never healthy. The empty reasonCodes is the discriminator: the sound plane emits three codes, the affected plane emits zero, and zero codes is currently indistinguishable from "all checks passed".

The two planes differed in revision at filing time (control a7b58e6a…, affected 2397b940…). (2026-08-21: both SHAs predate the #17376 history rewrite and no longer resolve on any clone — see the retarget note on the dropped diff AC below.)

Why this outranks its symptom: every other open item from that deployment concerns work not getting done. This one concerns a plane that would report itself healthy while holding no recoverable copy of the Memory Core. A deploy guide on that same plane instructs an operator to drop named volumes, with the reassurance that Memory Core and backups are untouched — that reassurance is read against this surface.

The Architectural Reality

  • maintenance.backup in the Memory Core healthcheck payload, and the same section in the orchestrator deployment-state bridge snapshot (snapshot.maintenance.backup, snapshot.maintenance.lastBackup).
  • staleAfterMs: 90000000 (25 h) bounds observation freshness — a stale-window pass appears to be reachable independently of whether any backup exists.
  • Reason codes observed on the healthy control: off-host-durability-unmet, backup-retry-exhausted, backup-never-succeeded.
  • snapshot.maintenance.lastBackup.offHostSync.exitCode exists in the snapshot schema, so off-host sync outcome is a carried field and a candidate for the missing link.

The Fix

  1. Determine whether maintenance.backup on the affected plane returns healthy from an absent observation or from an observation that genuinely evaluates to no reason codes. Establish which before changing the scorer. (Answered mechanically on PR #17475 by reproducing the affected plane's exact payload from the current head.)
  2. Make absent or unevaluated observations fail closed: an observation that cannot be read is unknown/degraded, never healthy. Do not let empty-reasonCodes stand in for verified-clean.
  3. A plane with zero backups must never read healthy — and the pre-first-run unanchored state stays pending, because a fresh deployment is not an incident and a warning that fires on every clean start costs the signal this field exists to carry (backup.mjs:312 records that decision). Only healthy is refused for zero-backup facts. (Retargeted 2026-08-21 — the original step demanded unconditional degradation, which degrades every fresh deployment.)
  4. Add the revision-diff outcome to the ticket once (1) is known. (Dropped 2026-08-21: both cited revisions died in the #17376 history rewrite — git cat-file -t a7b58e6a… fails on every clone — and step 1's payload reproduction at current head answered the question the diff would have.)

Contract Ledger Matrix

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
maintenance.backup.status Memory Core healthcheck Never healthy from an absent/unevaluated observation Existing enum unchanged Healthcheck schema docs healthy + reasonCodes: [] on a zero-backup plane
maintenance.backup.status on zero-backup facts same Never healthy; pre-first-run unanchored stays pending pending survives; only healthy refused same backup.mjs:312 — pending is the designed pre-first-run state
maintenance.backup.observationStatus same Verdict-evaluated status whose vocabulary is disjoint from the bridge-level field: observed / partial, never available / stale / degraded / unavailable; deliberately not required — a rolling deploy can pair a new Memory Core with an older orchestrator, and a required field would fail the healthcheck and gate tool admission Absent field tolerated Both schema descriptions cross-reference The available this ticket originally cited lives on maintenance.observationStatus, one level up — a bridge-reachability field, not a verdict field. The ticket had read the wrong level, which is this ticket's own defect shape applied to itself

Withdrawn row (2026-08-21): the original second row keyed a backup-never-succeeded veto on the inventory census (count === 0). Withdrawn rather than repaired: buildBackupStateBlock returns the same zero object for a backup path that does not exist and for a mount that is genuinely empty, and canonical mc-server carries no backup mount — a veto keyed on that census reads "cannot see the root" as "observed empty" and degrades every healthcheck there. Making the census vetoable needs a producer-owned observability contract on the census itself (HealthService's boundary) — successor ticket #17495.

Decision Record impact

none.

Acceptance Criteria

  • The affected plane's healthy verdict is traced to either an absent observation or a genuine empty-code evaluation, and the finding is recorded on this ticket.
  • A plane with zero backups (count === 0 && lastSuccessful === null) never reads healthy. The pre-first-run unanchored state remains pending; only healthy is refused. (Retargeted 2026-08-21 from "always yields degraded carrying backup-never-succeeded" — the literal form turns every fresh deployment into a never-succeeded incident before the lane has had one opportunity.)
  • An absent or unreadable backup observation resolves to unknown/degraded, never healthy.
  • maintenance.backup.observationStatus distinguishes observed-clean from not-observed, with a vocabulary disjoint from the bridge-level maintenance.observationStatus, and is not required in the schema.
  • Red-proof: a fixture with zero backups and a suppressed observation path must report healthy on main and degraded after the change. A fixture with a working observation path already reports degraded on main — as Ada's control plane demonstrates — so it would prove nothing.
  • The revision diff between a7b58e6a and 2397b940 is checked for an already-landed fix before implementation. (Dropped 2026-08-21: both SHAs died in the #17376 history rewrite and resolve on no clone; AC-1's mechanical answer at the current head makes the diff moot as well as unrunnable.)

Out of Scope

  • Making backups actually succeed on the affected plane, and off-host durability configuration. This ticket is the reporting surface only — a plane may legitimately have no backups, but it must say so.
  • Backup retention or verification policy.
  • The census observability contract (unmounted plane vs empty mount) — #17495.

Avoided Traps

  • Do not "fix the scorer". Ada's control proves the scorer already produces the right verdict from the same facts. Changing it without step (1) would be fixing a component that works and leaving the absent observation in place.
  • Do not treat an empty reasonCodes as evidence of health anywhere. That equivalence is the actual defect, and it will recur in any other derived status that adopts the pattern.
  • Do not key a veto on a census that cannot distinguish "unmounted" from "empty". That is this ticket's defect rebuilt one layer up — it is how the withdrawn ledger row failed, and #17495 owns the repair.
  • Do not degrade fresh deployments. pending is the designed pre-first-run state; a warning operators learn to ignore costs the signal.

Related

  • #17336, #17337 — the other two findings from the same deployment read; both concern work not happening, this one concerns a false assurance.
  • #17495 — successor for the census observability contract (same zero object for unmounted vs empty).
  • Ada's control run on neo-local-canonical at 2026-08-18T09:08:09Z supplied the falsifying comparison that reframed this from a derivation bug to a fail-open default.

Live latest-open sweep: checked latest 20 open issues at 2026-08-18T09:29:25Z plus a six-term state=all title sweep including backup status; no equivalent found.

Origin Session ID: 9ccc2fa1-8843-4796-8e85-5e151c0392d2

Retrieval Hint: query_raw_memories("backup status healthy empty reasonCodes zero backups fail-open observation")


Body retarget 2026-08-21 (author). Folded Ada's alignment comment into body authority after @neo-gpt's PR #17475 round-2 review flagged the drift: the zero-backup AC retargeted (pending survives; only healthy refused), the census-keyed veto row withdrawn → #17495, the observationStatus row corrected (disjoint vocabulary, not required, and the field-level mis-read fixed), and the dead-SHA diff AC dropped (#17376 rewrite). The comment is the dialogue; this body is the state.

tobiu referenced in commit bf224ba - "fix(ai): an unread backup observation stops scoring as a clean one (#17338) (#17475) on Aug 21, 2026, 11:57 PM
tobiu closed this issue on Aug 21, 2026, 11:57 PM