LearnNewsExamplesServices
Frontmatter
id16298
titleWAL drain locks survive container recreation as live PID 1
stateClosed
labels
bugairegressionarchitecture
assigneesneo-gpt-emmy
createdAtAug 1, 2026, 9:58 PM
updatedAtAug 1, 2026, 10:48 PM
githubUrlhttps://github.com/neomjs/neo/issues/16298
authorneo-gpt-emmy
commentsCount0
parentIssue14477
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtAug 1, 2026, 10:48 PM

WAL drain locks survive container recreation as live PID 1

Closed Backlog/active-chunk-11 bugairegressionarchitecture
neo-gpt-emmy
neo-gpt-emmy commented on Aug 1, 2026, 9:58 PM

Context

The one-time D#15758 cohort rebuild started a replacement Memory Core container at 2026-08-01T19:39:47Z. Both persisted drain locks still recorded the previous container's PID 1 and 13:45:19Z epoch. Startup then logged both in-process loops as NOT started.

Four later add_memory writes produced pendingDrainDepth: 4; the oldest age rose past six minutes with no decrease. Recency reads still worked, but semantic projection had stopped. The message-WAL loop failed for the same reason.

This is a linked successor to the resolved lock work, not a reopen.

The Problem

The locks live inside persistent WAL volumes, but their default liveness predicate is only process.kill(holder.pid, 0). After container recreation, the new container also has a live PID 1. The probe therefore mistakes a dead previous boot's owner for the current process and refuses the only drainer.

A healthy MCP process can consequently serve reads while every new memory remains semantically pending and message projection stops.

The Architectural Reality

  • ai/daemons/embed/drainLock.mjs:11-15,107 claims the contenders are always same-namespace and delegates freshness solely to PID liveness.
  • ai/daemons/message/drainLock.mjs wraps that same primitive, so one correction covers both WAL families.
  • ai/daemons/shared/fileLease.mjs already supports additive descriptor fields, opaque owner tokens, guarded pulse/release, and an injected freshness predicate.
  • #16262 / PR #16266 established the sibling solution: boot identity handles cross-boot PID reuse; process-start time handles a legacy same-PID prior epoch.
  • The mandatory structure-map command was run and failed with Cannot create a string longer than 0x1fffffe8 characters. Direct import/consumer inspection confirms the existing drain-lock specialization is the owning substrate; no new file is needed.

The Fix

Extend acquireDrainLock() without changing its public placement:

  1. Write additive bootId lease metadata, defaulting to os.hostname() and injectable in tests.
  2. Treat a present, different bootId as stale before PID liveness.
  3. Treat an equal PID whose startedAt predates the current process start as a legacy previous epoch. This reclaims today's pre-bootId PID-1 lock on the first fixed restart.
  4. Preserve fail-closed behavior for pre-field, different-PID holders that still appear live.

The message wrapper inherits the corrected semantics.

Contract Ledger

Target surface Authority Behavior Fallback Docs Evidence
acquireDrainLock(options) descriptor + freshness ai/daemons/embed/drainLock.mjs over shared acquireFileLease additive bootId; cross-boot mismatch and legacy same-PID prior epoch reclaim absent bootId + different live PID keeps current refusal module/JSDoc option contract focused embed + message drain-lock specs

Decision Record impact

aligned-with ADR 0014 and aligned-with ADR 0019. This changes no task ownership or config leaf.

Acceptance Criteria

  • A different-bootId PID-1 lock is reclaimed even when the PID probe returns alive.
  • A pre-field, equal-PID lock older than the current process start is reclaimed.
  • A same-boot live holder still refuses takeover.
  • A pre-field, different live PID keeps existing fail-closed behavior.
  • Both memory and message drain-lock suites prove the inherited behavior.
  • [L3-deferred] After rollout, both loops start and the observed backlog drains to zero without manual lock deletion.

Out of Scope

Deployment automation; authority/heavy-maintenance lease changes; changing WAL semantics; blindly deleting live-plane lock files.

Avoided Traps

  • Blind lock deletion: can violate the sole-drainer invariant.
  • PID equality as ownership: numeric PID reuse is the defect.
  • TTL-only takeover: can either delay recovery or steal a valid long drain; boot/process epochs are the direct discriminator.

Related

Parent: #14477

Related: #15802

Related: #16210

Related: #16262

Related: #16167

Related: PR #16266

Related: D#15758

Duplicate Sweep

Live latest-open sweep: latest 20 open issues checked at 2026-08-01T19:57Z; no equivalent. The 30-message all-state A2A claim sweep found no overlapping claim. KB ticket search surfaced parent #14477 and closed predecessor #15802, not an active leaf. Exact repository search found only the #16210/#16262 precedent chain.

Origin Session ID: 019fb600-58b9-7fa2-86a7-5a15e1ccf659

Retrieval Hint: WAL drain lock container recreate PID 1 bootId pendingDrainDepth

tobiu referenced in commit 0c4a09b - "fix(agent-os): distinguish WAL drain boot epochs (#16298) (#16299)" on Aug 1, 2026, 10:48 PM
tobiu closed this issue on Aug 1, 2026, 10:48 PM