LearnNewsExamplesServices
Frontmatter
id12394
titleSchedule GraphLog compaction as a recurring Orchestrator maintenance lane
stateClosed
labels
enhancementaiarchitecture
assigneesneo-gpt
createdAtJun 2, 2026, 9:17 PM
updatedAtJun 7, 2026, 7:18 PM
githubUrlhttps://github.com/neomjs/neo/issues/12394
authorneo-opus-ada
commentsCount1
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
blockedBy[x] 12329 GraphLog grows unbounded: no compaction past consumer watermark
blocking[]
closedAtJun 3, 2026, 12:06 PM

Schedule GraphLog compaction as a recurring Orchestrator maintenance lane

Closed Backlog/active-chunk-17 enhancementaiarchitecture
neo-opus-ada
neo-opus-ada commented on Jun 2, 2026, 9:17 PM

Context

#12329 established that GraphLog (the Native Edge Graph CDC table) grows monotonically and unbounded during normal operation — the live production read in #12329 found it at 7.3M rows / 365 MB (83% of the graph db). #12337 delivered the compaction mechanism: ai/scripts/maintenance/compactGraphLog.mjs, a dry-run-first CLI that prunes rows at/below the minimum live-consumer watermark (#12389 fixes its standalone-Node bootstrap).

But that is a manual CLI. #12329's ACs delivered the mechanism (AC2: "a compaction mechanism … runnable periodically"); none of them schedules it. So the CDC log only stays bounded if a human remembers to run the CLI — and resumes unbounded growth between runs. For a continuously-appended CDC table, that is the recurring-job-shipped-as-a-one-shot gap.

The Problem

GraphLog compaction is recurring maintenance with no automation. The Agent OS already owns a periodic-maintenance scheduler — the Orchestrator (ADR 0014) — which runs sibling store-maintenance lanes (backup, chromaDefrag, summary, dream, golden-path). GraphLog compaction is the missing sqlite-graph-store equivalent of the existing chromaDefrag (chroma-store) lane. Cloud deployments are more exposed, not less: their orchestrator runs the graph-mutating dream/summary lanes, so their GraphLog grows the same way, with no operator at a shell to run the CLI.

The Architectural Reality

  • Scheduler substrate: ai/daemons/orchestrator/TaskDefinitions.mjs (buildTaskDefinitions — the lane→child-process command map; e.g. backupmaintenance/backup.mjs) + ai/daemons/orchestrator/scheduling/*.mjs (per-lane getDueTask cadence) + Orchestrator.mjs (the lane registry + cloud-safe deployment-policy classification, e.g. tenant-repo-sync cloud-deployable vs primary-dev-sync local-only).
  • The job to invoke: ai/scripts/maintenance/compactGraphLog.mjs (--apply; the watermark-safety contract — never prune above min(live-consumer watermark) minus DEFAULT_SAFETY_MARGIN_ROWS — already lives in the CLI per #12329 AC3, so the lane delegates safety to it).
  • Precedents to mirror: backup (low-frequency, cloud-deployable, retention-bounded) and chromaDefrag (the chroma store-maintenance lane). This ticket adds the symmetric graphlog-compaction lane.

The Fix

Add a graphlog-compaction Orchestrator lane:

  1. TaskDefinitions.mjs — a lane entry invoking maintenance/compactGraphLog.mjs --apply (with --vacuum per cadence/config), pidfile, expected-command guard, mirroring backup.
  2. scheduling/graphLogCompaction.mjs — a getDueTask with a low-frequency cadence (config-driven, default daily/weekly like backup).
  3. Orchestrator.mjs — register the lane + classify it in the cloud-safe deployment policy (it should run in cloud deployments — their GraphLog grows too).
  4. Cadence/enable knobs as config leaves (sibling to the backup retention config).

Contract Ledger

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
TaskDefinitions.buildTaskDefinitions graphlog-compaction lane this ticket child-process command invoking compactGraphLog.mjs --apply absent → no scheduled compaction (today's state) TaskDefinitions JSDoc Orchestrator.spec lane-definition assertion
scheduling/graphLogCompaction.mjs getDueTask this ticket + scheduling/backup.mjs precedent low-frequency due-check (config cadence) unset cadence → conservative default scheduling JSDoc unit: due/not-due across the cadence window
Cloud-safe deployment-policy classification ADR 0014 + Orchestrator.mjs policy graphlog-compaction runs in cloud deployments local-only opt-out if a deployment disables it deployment-policy comment Orchestrator.spec classification assertion
Watermark-prune safety compactGraphLog.mjs (#12329 AC3) unchanged — delegated to the CLI the CLI fail-closes on unknown watermarks n/a (existing) existing compactGraphLog.spec

Decision Record impact

aligned-with ADR 0014 (Cloud Deployment Topology + Scheduler Task Taxonomy) — extends the orchestrator lane taxonomy with the symmetric graph-store maintenance lane; does not challenge it.

Acceptance Criteria

  • A graphlog-compaction Orchestrator lane exists (TaskDefinitions entry + scheduling/ getDueTask) that periodically invokes compactGraphLog.mjs --apply.
  • The lane is classified per the cloud-safe deployment policy and runs in cloud deployments (verifiable in Orchestrator.spec).
  • Cadence + enable are config-driven (sibling to the backup config), with a conservative default.
  • Watermark-prune safety is delegated to the existing CLI (no duplicated safety logic).
  • Tests: lane-definition + deployment-policy assertions in Orchestrator.spec; due/not-due unit for the new scheduler.

Out of Scope

  • The compaction mechanism itself (compactGraphLog.mjs) — delivered by #12329 / #12337; bootstrap by #12389.
  • The one-time existing-backlog purge (operator-gated; #12329 AC5).
  • The upstream mutation-churn source (REM-cycle / GapInferenceEngine tuning) — a separate concern noted on #12329.

Related

  • #12329 (GraphLog unbounded-growth root + the compaction mechanism) · #12337 (CLI) · #12389 (CLI bootstrap fix — this lane is blocked by the CLI booting in a standalone Node process).
  • chromaDefrag / backup orchestrator lanes (the maintenance-lane precedent).

Origin Session ID: c279991f-11eb-4e89-9e38-2c9c4a78421e

Handoff Retrieval Hints: query_raw_memories("GraphLog compaction orchestrator lane recurring maintenance chromaDefrag precedent"); anchors ai/daemons/orchestrator/TaskDefinitions.mjs, ai/scripts/maintenance/compactGraphLog.mjs.

Authored by @neo-opus-ada (claude-opus-4.8-1m)

tobiu referenced in commit d50e9d7 - "feat(orchestrator): schedule graphlog compaction (#12394) (#12395) on Jun 3, 2026, 12:06 PM
tobiu closed this issue on Jun 3, 2026, 12:06 PM