Per-append exclusive write-lock on the shared WAL segment — never-fail-preserving (Part 2 of embed-drain recovery)

Context

Part 2 of the embed-drain recovery (Refs #13495). Root cause (diagnosed jointly with @neo-opus-grace): non-github harness clones' add_memory WAL records orphan — the single embed-drainer (sole-drainer invariant, #12864) only sees the github clone's per-clone WAL, so the other 6 clones' records WAL-write durably but never embed → un-searchable (~2.5k records swarm-wide).

The converged fix (operator-validated, chosen over a multi-dir-drainer refactor of the critical drain path): unify the WAL into one shared dir so the single drainer sees every clone's records. BUT the WAL segment is one-writer-per-file by design (memoryWalStore.mjs:14-23): "POSIX O_APPEND atomicity is only dependable for small writes, and memory records (multi-KB thought payloads) appended concurrently with tiny markers from a second process could interleave." So a shared dir requires the writers be serialized.

The Fix

A per-append exclusive write-lock in appendWalMemory (ai/services/memory-core/helpers/memoryWalStore.mjs), so the clones' MC-server processes serialize their appends to the shared wal-YYYY-MM-DD.jsonl. Mirror the existing #12864 ai/daemons/embed/drainLock.mjs wx-lockfile pattern (atomic wx create + PID/mtime stale-reclaim) — dependency-free (fs-ext/flock is not a dep).

Acceptance Criteria

• appendWalMemory acquires a per-segment-file exclusive lock (wal-DATE.jsonl.lock, atomic wx) before the append, releases after — serializing concurrent multi-KB appends so they can't interleave.
• Never-fail (nail #1): a failed/timed-out acquire lets add_memory fall through and write anyway (bounded acquire-timeout) — NEVER hangs the turn-save. The lock is best-effort serialization, not a gate on the durable write.
• Stale-reclaim: a lock held by a dead PID (or past a TTL) is reclaimed, so a crashed clone can't block others' add_memory (mirrors drainLock's reclaim).
• Writer-vs-writer only: the drainer keeps reading lock-free (tolerating a trailing partial line, as it already does) — no drainer or file-format change.
• Unit coverage: contention (two writers serialize, no interleave), stale-reclaim (dead-PID lock reclaimed), never-fail (acquire-timeout → falls through + still writes).

Sequencing (critical)

This must land + roll out to every clone's MC before Part 1 (the shared-dir symlink flip) — an un-updated clone would write unlocked and race. Then Part 3 (recovery) + the github drainer embeds. So this is the blocking-first step.

Out of Scope

• Part 1 (bootstrapWorktree.mjs whitelist→blacklist + the shared-dir symlink) — @neo-opus-grace.
• Part 3 (ephemeral one-shot recovery: orphaned WALs → github, dedup by UUID) — @neo-opus-grace + @tobiu.
• The WAL-backlog liveness-watchdog (recurrence guard) — my follow-up (the silent ~8-day orphaning had no alarm).

Related

Refs #13495 (embed-drain recovery thread). Mirrors #12864 (drainLock). Aligns with the never-fail WAL design (#12838 / #12840).

Release classification: incident-recovery (v13.1 agent-OS) — boardless.

Origin Session ID: a200ff3b-5ea5-4fc0-b288-cd5fe2c917e0

Authored by Claude Opus 4.8 (Claude Code), @neo-opus-vega (Vega).