LearnNewsExamplesServices
Frontmatter
id16837
titleRecovery audit sinks lose write-time authority provenance
stateClosed
labels
bugaitestingarchitectureagent-os
assigneesneo-gpt-emmy
createdAtAug 9, 2026, 10:47 PM
updatedAtAug 10, 2026, 3:19 AM
githubUrlhttps://github.com/neomjs/neo/issues/16837
authorneo-gpt-emmy
commentsCount0
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtAug 10, 2026, 3:19 AM

Recovery audit sinks lose write-time authority provenance

Closed Backlog/active-chunk-14 bugaitestingarchitectureagent-os
neo-gpt-emmy
neo-gpt-emmy commented on Aug 9, 2026, 10:47 PM

Context

PR #16778 at exact head b43ebfccdde5ac8d1540d3f00a9f108abb4343b6 closes the privileged-effect and durable-ledger authority fences required by #16766: provider mutations stop after authority loss, recordDiagnosis() revalidates before its second append, dispatched effects retain an explicit heldAtWrite: false audit, and scratch writes use globally unique identity plus adjacent fences.

The merge-safe remainder belongs in a new successor rather than a twenty-second repair comment. The privileged effects are fenced and the local recovery-run JSONL record is truthful; the residual defect is that two derived/audit proof surfaces can lose that write-time provenance without authorizing another effect.

Live latest-open sweep: checked the latest 20 open issues created through 2026-08-09T20:21:18Z immediately before filing; no equivalent ticket exists. The recent A2A claim slice had no overlapping lane. Repository issue/discussion search found only the #16766 implementation history. Knowledge Base retrieval surfaced adjacent graph-reseed and generic TTL-writer tickets, but no ticket for this recovery authority-proof composition.

[TOOLING_GAP] The mandatory Agent OS structure-map command was attempted before filing:

npm run --silent ai:structure-map -- --files --loc
Cannot create a string longer than 0x1fffffe8 characters

No new file or placement is proposed; the existing sibling owners remain recoveryRunStateStore.mjs and healEventLedgerStore.mjs.

The Problem

1. The recovery-run file and graph disagree

At the exact head, appendRecoveryRunState() samples authority immediately before the JSONL append and creates a stamped record. It writes that record to disk, then publishes the original unstamped entry:

  • ai/services/memory-core/helpers/recoveryRunStateStore.mjs:528-546
  • createRecoveryRunGraphProperties():619-645 does not project heldAtWrite

A production-shape exact-head probe with isAuthorityHeld() === false and a genuinely dispatched effect produced:

{
  "fileHeldAtWrite": false,
  "graphHeldAtWrite": null,
  "graphDetails": {"heldAtAppend": true},
  "nodeCount": 3
}

The local durable audit is correct. The remotely readable graph proof is not merely incomplete: it can retain the older details.heldAtAppend: true while omitting the later authoritative sample. Consumers can therefore see the wrong provenance depending on which proof surface they read.

2. The first record-only sink has no adjacent authority sample

RecoveryActuatorService.recordDiagnosis() checks authority at method entry, then awaits appendHealEvent(). The helper awaits mkdir() before appendFile() and accepts no authority oracle:

  • ai/daemons/orchestrator/services/RecoveryActuatorService.mjs:547-612
  • ai/services/memory-core/helpers/healEventLedgerStore.mjs:101-113

A takeover during that awaited directory step can admit a shared status: 'recorded' row after the predecessor lost authority. This path dispatches no privileged lifecycle effect, so it is merge-safe follow-up scope; it still needs truthful write-time provenance because ADR 0026 defines this ledger as the durable async-audit terminal.

The exact-head controls also lag the repaired production paths: the default provider-helper test recreates the assertion in a test twin, the reconfigure-named control calls restart, and no raise-ceiling takeover control drives the real oracle-bearing path.

The Architectural Reality

The recovery-run JSONL append and its graph publication are one source/projection operation. Once the store accepts a stamped record, the graph projection must publish that same record; it must not sample authority again and create a second decision after the source append.

The heal-event ledger is shared by several non-lease callers, so its existing behavior must remain compatible when no oracle is supplied. The recovery actuator needs an optional store-adjacent authority/provenance path: sample after awaited setup and immediately before the append, then refuse an audit-only owner-success write after loss or stamp the admitted record truthfully according to the caller contract.

This composes with ADR 0026 AC-6 (durable record-with-diagnosis) and ADR 0024's source/projection discipline. It does not amend either decision.

The Fix

  1. Publish the exact stamped recovery-run record to the graph.
  2. Add heldAtWrite to recovery-run graph properties and the normalized remote proof record. Legacy absence is unknown, never true or false.
  3. Give the heal-event append boundary an optional authority/provenance seam suitable for recordDiagnosis(), sampled after awaited setup and adjacent to appendFile(). Preserve byte-compatible behavior for callers that supply no oracle.
  4. Bind takeover controls to the real production paths: default LMS/Ollama helpers, the transition between the two record-only sinks, actual reconfigure, and actual raise-ceiling.
  5. Keep graph publication derived from the accepted source record. Do not re-check the lease after the JSONL append and thereby split source and projection.

Contract Ledger Matrix

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
Recovery-run JSONL heldAtWrite Store-adjacent isAuthorityHeld sample Existing truthful Boolean stamp remains unchanged No oracle means unknown/legacy-compatible Existing helper JSDoc Exact-head displaced-effect probe
Recovery-run graph properties.heldAtWrite The already accepted stamped JSONL record Project the same Boolean/null provenance; normalized reads preserve it Legacy missing field normalizes to unknown Update projection JSDoc File↔graph parity controls
Heal-event record-only append Optional store-adjacent authority sample A displaced owner cannot write an unqualified recorded row; admitted rows carry truthful provenance Callers without an oracle keep current behavior Update helper and actuator JSDoc Takeover-during-setup control
Reconfigure / raise-ceiling / provider mutation controls Real production oracle threading Tests drive the actual effect paths and prove zero mutation after loss N/A Test titles state the exercised action Production-bound mutation controls

Acceptance Criteria

  • A displaced dispatched-effect record yields heldAtWrite: false in both JSONL and every recovery-run graph proof node that represents that state.
  • heldAtWrite: true, false, and legacy/unknown behavior are independently covered; details.heldAtAppend cannot override or substitute for the store-adjacent field.
  • recordDiagnosis() carries authority to the heal-event store boundary; takeover during awaited setup cannot produce an unqualified owner-success audit row.
  • Existing heal-event callers without an authority oracle retain their current append contract.
  • Production-bound takeover controls cover the default provider helpers, the interval between the two record-only sinks, actual reconfigure, and actual raise-ceiling.
  • Mutation proofs demonstrate that removing each production fence/projection step turns the named control red.
  • Graph publication does not introduce a second post-append lease decision that can diverge from the accepted source record.
  • Focused Memory Core helper and Orchestrator unit suites pass at the implementation head.

Out of Scope

  • Reopening or extending #16766.
  • Adding recovery action classes, widening runtime privilege, or changing the anti-thrash policy.
  • Re-deciding ADR 0026 or ADR 0024.
  • Making every heal-event caller lease-aware.
  • Treating graph publication as a second authority-bearing mutation after the JSONL source append.

Avoided Traps

  • Stale provenance as fallback: details.heldAtAppend is an earlier sample and cannot stand in for heldAtWrite.
  • Double authority: re-sampling after the accepted JSONL append can make source and projection disagree in the opposite direction.
  • Globalizing one caller's lease: unrelated heal-event producers do not inherit orchestrator lease semantics.
  • Test twins: recreating an assertion inside a stub can stay green while the production helper loses its fence.

Related

Refs #16766
Related: #16778

Decision Record impact

Aligned with ADR 0026 AC-6 and ADR 0024 source/projection discipline; no amendment.

Origin Session ID: 878f05af-2c4e-4da2-a5c2-9e4af666fcb8

Retrieval Hint: PR #16778 b43ebfcc heldAtWrite recoveryRunStateStore healEventLedgerStore authority projection

tobiu referenced in commit fc7a9df - "fix(recovery): preserve write-time audit provenance (#16837) (#16848) on Aug 10, 2026, 3:19 AM
tobiu closed this issue on Aug 10, 2026, 3:19 AM