REM run JSONL store: retention/prune to bound file count + read cost

Context

PR #12122 (Resolves #12088, Part B of #12068 under Epic #12065) shipped RemRunStateStore.mjs — a durable per-cycle REM run/stage JSONL state model writing one <runId>.jsonl artifact per REM cycle to remRunStateDir (default .neo-ai-data/rem-runs/).

Surfaced during cross-family /pr-review of PR #12122 (review PRR_kwDODSospM8AAAABBO9n5w, Approve+Follow-Up). The PR author (@neo-gpt) confirmed the retention gap is known future-work — their PR Slot-Rationale decay-mitigation explicitly anticipates "a later retention policy supersedes the JSONL operator guidance". This ticket tracks that follow-up.

The Problem

RemRunStateStore ships appendRemRunState (write) + readRecentRemRunStates (read) but no prune/rotation. Two compounding effects accumulate with deployment age (NOT with the bounded read window):

1. Unbounded file count — one <runId>.jsonl per REM cycle, never removed. At the default 1h dream cadence (orchestrator.intervals.dreamMs) that is ~24 files/day; the directory grows without bound across a deployment's lifetime.
2. Read-path cost grows monotonically — readRecentRemRunStates performs fs.readdir(dir) then fs.stat on every .jsonl file on every call (to mtime-sort before applying remRunRecentLimit). The remRunRecentLimit config bounds how many entries are RETURNED, but not how many files are scanned. Every get_rem_pipeline_state MCP/healthcheck call therefore pays N stat syscalls where N = total accumulated run-files, not the recent-limit window.

The feature is correct today; this is observability-debt that degrades over months of continuous operation — exactly the cloud-deployment posture the REM substrate targets.

The Architectural Reality

• ai/services/memory-core/helpers/RemRunStateStore.mjs
  • appendRemRunState(entry, {dir}) (~line 134) — fs.mkdir(recursive) + fs.appendFile; no cap, no prune of prior artifacts.
  • readRecentRemRunStates({dir, limit}) (~line 155) — fs.readdir → fs.stat per .jsonl → mtime-sort → slice(limit). The per-file stat fan-out is the cost that scales with directory size.
• ai/mcp/server/memory-core/config.template.mjs — remRunStateDir + remRunRecentLimit (read-window bound only).
• Consumer: get_rem_pipeline_state MCP tool projects recentCycles via readRecentRemRunStates (operator-facing healthcheck surface).

The Fix

Single prescription: add a retention mechanism that bounds the on-disk artifact set. The exact mechanism (write-side cap vs. dedicated cleanup lane) is an empirical decision gated by an isolation test (AC1), not pre-committed here — both are viable substrate locations and the perf-curve should drive the choice:

• Option A — write-side cap: appendRemRunState prunes artifacts beyond a configurable retention bound (age- or count-based) on each write. Keeps retention co-located with the store; no new lane.
• Option B — orchestrator cleanup lane: a periodic orchestrator task sweeps remRunStateDir against a retention policy. Decouples retention cadence from write cadence; consistent with the cloud-safe scheduler taxonomy (ADR 0014).

Service-boundary note: Option A keeps the concern inside the memory-core helper that owns the store; Option B introduces an orchestrator-owned lane consuming the memory-core store across the service boundary. The isolation test (AC1) should surface whether the read-path cost justifies the cross-boundary lane or whether a write-side cap suffices.

Contract Ledger

Acceptance Criteria

• AC1 (empirical isolation test — decides the mechanism): a test seeds the run-state dir with N artifacts where N substantially exceeds remRunRecentLimit, then measures readRecentRemRunStates behavior (readdir/stat fan-out). The measured read-path cost curve informs the write-side-cap vs cleanup-lane choice; the chosen mechanism is documented in the PR with the isolation-test evidence.
• AC2 (retention bounds the artifact set): after the retention mechanism runs, the on-disk artifact count stays within the configured retention bound; older artifacts beyond the bound are removed. Verified by a unit test exercising the prune path.
• AC3 (config-driven, no hidden fallback): retention is governed by a config knob with a concrete default in config.template.mjs + env binding; code reads the config verbatim (operator config-SSOT contract; no ||/?? substitution or resolver-derive).
• AC4 (read-path no longer scales with deployment age): with retention active, readRecentRemRunStates cost does not grow unbounded with total cycles run. Architectural-layer assertion (the read fan-out is bounded by retention, not by lifetime cycle count) — not a hardcoded latency cap.
• AC5 (operator doc): learn/agentos/rem-state-model.md documents the retention policy + the chosen mechanism + the disabled-mode degraded behavior.
• AC6 (no observability regression): get_rem_pipeline_state.recentCycles still returns the most-recent cycles correctly after retention runs (retention removes oldest, never the recent window).

Out of Scope

• Changing the JSONL state-model schema or the recentCycles projection shape (PR #12122 contract is stable).
• The 5-axis observability primitive (#12068 Part A, shipped).
• Per-phase wall-clock / cadence-overflow detection (#12088 scope, shipped in PR #12122).
• Compaction of individual JSONL files (each is single-entry per run; the concern is file COUNT, not intra-file size).

Avoided Traps

• ❌ Hardcoding a fixed file-count cap in an AC — the right bound is deployment-policy-dependent; the empirical isolation test (AC1) + a config knob (AC3) is the substrate-correct shape, not a magic number baked into the AC.
• ❌ Defaulting to the orchestrator cleanup lane without measuring — a write-side cap may suffice and avoids a cross-service-boundary lane; the isolation test decides.
• ❌ Graph-node-based retention tracking — the JSONL store is deliberately filesystem-durable (per #12088 Avoided Traps: graph mutations during REM run risk active-control-plane collateral damage); retention stays in the filesystem layer.

Decision Record impact

none — enhancement to a just-shipped primitive; no ADR conflict. (ADR 0014 cloud-scheduler taxonomy is aligned-with IF Option B cleanup-lane is chosen, but that is an AC1-gated implementation detail, not an ADR amendment.)

Related

• Parent context: Epic #12065 (Orchestrator-as-SSOT for REM Pipeline); sibling of #12088 (Part B, shipped via PR #12122).
• Surfacing PR: #12122 — cross-family review Approve+Follow-Up where this gap was identified + author-confirmed.
• Discussion lineage: #12062 §2.11 (REM state-model schema).

Origin Session ID

3e21265b-38a8-4bf3-a81f-04375d8a757f

Handoff Retrieval Hints

query_raw_memories: "REM run JSONL retention prune RemRunStateStore readdir stat per file unbounded". Git anchor: PR #12122 RemRunStateStore.mjs appendRemRunState / readRecentRemRunStates.