LearnNewsExamplesServices
Frontmatter
id17413
titleThe embedding lane has no end-to-end description, so its competing mechanisms are only discoverable by re-measuring the plane
stateClosed
labels
documentationai
assigneesneo-opus-vega
createdAtAug 20, 2026, 11:32 AM
updatedAtAug 21, 2026, 9:15 PM
githubUrlhttps://github.com/neomjs/neo/issues/17413
authorneo-opus-vega
commentsCount0
parentIssue17411
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[x] 17412 The embedding dispatch loop awaits every request, so a lane declaring four parallel slots runs one
blocking[]
closedAtAug 21, 2026, 9:15 PM

The embedding lane has no end-to-end description, so its competing mechanisms are only discoverable by re-measuring the plane

Closed Backlog/active-chunk-17 documentationai
neo-opus-vega
neo-opus-vega commented on Aug 20, 2026, 11:32 AM

Context

ADR 0019 §1 states the mechanism this ticket applies elsewhere:

"The empirical lesson is not 'be more careful' — that is falsified (4/4 missed across 2 doc-prepared reviews). It is the broken-window root: a pattern-matcher with no grasp of the sanctioned mechanism has only the broken code to match. This ADR is that mechanism, made readable."

The embedding lane has no such artifact. learn/agentos/cloud-deployment/TenantIngestionModel.md is the most detailed document on the subsystem — 492 lines, zero diagrams — and covers ingestion configuration, triggers, and telemetry rather than the lane itself. learn/agentos/KnowledgeBase.md is the narrative tier. Neither describes what happens between "a chunk exists" and "a vector is stored".

The Problem

The lane has no owning document because it has no owning subsystem. TextEmbeddingService.mjs lives under ai/services/memory-core/, VectorService.mjs under ai/services/knowledge-base/, the geometry leaves in ai/configBase.mjs, the slice/lease scheduling in ai/daemons/orchestrator/, and the shape verification in ai/providerLaneLiveShape.mjs. Four owners, no owner — so no document is anybody's to write, and the composition is only discoverable by instrumenting a live plane.

That cost is now measured. Establishing the lane's actual behaviour on 2026-08-20 took a full session of plane reads to reach conclusions that were partly already recorded elsewhere, and produced four wrong intermediate claims before converging:

claim made corrected by
envelope materialization consumes the slice budget the provider slot log — one 13,725-token input did
the 32 GiB ceiling is unjustified the deployment's own measured derivation (24.14 GiB at 13,980 tokens)
fused attention is an unexplored win already attempted, no reduction, documented
the shape verification compares two declarations it reads live /slots; the verification is real

By session end the count was nine wrong intermediate claims against one genuinely novel finding. The other eight were each answerable from content already in the repository:

claim where the answer already lived
the memory ceiling is unjustified the deployment's compose comment — measured KV formula plus a 24.14 GiB peak at 13,980 tokens
fused attention is unexplored the same block — already attempted, no reduction, no log line
the shape verification compares two declarations that module's own docstring — it reads live /slots
the checkpoint deadlock is a new finding an open AC on #16566, written two weeks earlier
vendored trees need path exclusion built VENDOR_PATH_SEGMENTS already implements it in the tenant parser
the orphan-purge mechanism needs building merged in PR #17395
envelope materialization consumes the slice the provider slot log
a failure class does not occur on that plane judged from a 120-line log tail

The failure mode is not diligence — it is that no artifact presents the lane as one system, so every investigator rebuilds the map from symptoms and gets a different fraction of it. The distributed answers sit in a 492-line prose doc, a deployment compose comment, an ADR, a ticket AC, a parser docstring, and a merged PR.

The Architectural Reality worth drawing

The verification layer is sound and its own doc says why:

"liveness answers 'can the provider respond?', never 'is the provider shaped the way this deployment intends?'. This module supplies the second answer as data."

providerLaneLiveShape compares declared intent against a live /slots reading, is pure by construction, and refuses to resolve its own values because "a helper that resolved its own values would be a second authority able to disagree with the config".

The gap is one axis further out: shape is verified, utilization is not. The lane truthfully reports four slots at 16,384 tokens each, and nothing asks whether they are used. Measured on the plane: ~20 consecutive launch → release → launch transitions with never two tasks in flight, while 86,946 chunks waited. A guide that draws the verified axes beside the unverified one makes that omission visible instead of discoverable.

Two kinds of truth, and why that shapes the artifact

A ground-truth guide that goes stale is worse than no guide. Written one day before this session, it would have stated "the tenant parser declines vendored trees"true of the contract, false of the plane, which still carried ~86.7k vendored chunks materialized under the previous parser. A reader trusting it would have concluded the corpus was clean.

So the guide separates two classes of fact with different decay rates, and expresses them differently:

  • Contract facts — what the code does, the measured constants, the geometry arithmetic, the field names, the state machine. These change only when someone edits the code, so they are written down, with the citation that lets a reader re-derive them.
  • Plane facts — what is deployed, what is materialized, which signal has actually fired. These change without anyone editing anything, so each is written as a dated snapshot plus the command that re-derives it: deployedRevision versus origin/dev, git merge-base --is-ancestor <fix> <deployed>, the per-repo corpusOutstanding.

Both halves are load-bearing, and an earlier draft of this ticket got it wrong by demanding the command instead of the value. A guide is a snapshot by nature and will go stale; the goal is not to avoid stating what decays but to make the delta cheap. A reader who knows what the value was on a named date, and how to re-read it, can compute what changed in one command. A reader given only the command starts from zero — which is the cost this ticket exists to remove.

So the rule is: a plane fact carries its observation date and its re-derivation command; a plane fact stated as a timeless present is the defect. "The parser declines vendored trees" is a contract fact and belongs written plainly. "The corpus contains no vendored chunks" is a plane fact, was false while the contract sentence was true, and belongs dated with its query.

The Fix

A guide at learn/agentos/EmbeddingLane.md — sibling of KnowledgeBase.md, MemoryCore.md, SelfHealing.md, the diagram-bearing tier — describing the lane end to end, with the diagrams carrying the load rather than prose.

Scope is deliberately limited to what this session measured, so nothing in it is reconstructed from reasoning:

  1. Authority map — the geometry leaves, their env bindings, their consumers, with derivation edges drawn distinctly from independent declarations.
  2. One slice, as it actually runs — four slots, one request in flight, a 13,725-token input holding the lane, embeddings=1 at budget expiry.
  3. Checkpoint states — with the edge that cannot be traversed: lastIngestedRev is written only by a pass that completes, and a corpus larger than one slice never completes one.
  4. Guard interaction — the pairs whose joint outcome differs from either alone: slice-budget × checkpoint, admission-band × provider tokenizer, starvation-watchdog × lease-holder.
  5. Verified vs unverified axes — what providerLaneLiveShape establishes, and that utilization sits outside it.
  6. Code locality — the four-owner map, i.e. the ownership vacuum this ticket exists because of.

Contract Ledger Matrix

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
learn/agentos/EmbeddingLane.md (new) the measured plane reads on #17411 + the code it cites end-to-end lane description, six diagrams this IS the doc 492-line sibling with zero diagrams
learn/agentos/cloud-deployment/TenantIngestionModel.md unchanged gains a pointer; its ingestion-config scope is not duplicated cross-link only overlap would be accretion
learn/tree.json guide registry new entry guides are registry-listed
buildScripts/docs/seo/generate.mjs PRIORITIES the guide-cluster ranking map agentos/EmbeddingLane at 0.8 absent key falls back to DEFAULT_PRIORITY comment beside the entry states the ranking rationale registration is TWO inputs, not one — the tree entry alone leaves the page at the default rank
pipeline-owned sitemap / llms outputs the docs pipeline untouched generated artifacts; a hand-edit would be overwritten and would mask the generator's own behaviour
merge order this ticket #17413 lands after #17412 (PR #17433) recorded on both PRs the guide's concurrency/width/carry statements describe code that #17412 changes; a guide merged first would ship contract that is true of neither tree for as long as the gap lasts

Decision Record impact

aligned-with ADR 0019 — same remedy shape (the mechanism made readable), applied to a subsystem the ADR does not cover. No decision is changed. One ADR-0019 challenge is recorded as out of scope below.

Acceptance Criteria

  • Every number in the guide is either measured or cited to the code that computes it. No figure appears without a source; a reader can re-derive each one. Asserted by review against the cited surfaces, not by word count.
  • The band arithmetic is stated as arithmetic: admissionCeilingTokens = min(contextLimitTokens, safeProcessingLimitTokens), estimateBandTokens = floor(ceiling / 1.35), bytesToTokens = ceil(bytes / 3) — so a reader can compute the byte-per-token ratio at which admission overflows the slot (≈2.22) rather than being told a conclusion.
  • Diagram 4 names at least three guard pairs whose joint outcome differs from either alone, each with the observed consequence, so "layers compete" stops being an assertion.
  • Diagram 5 marks the verified/unverified boundary explicitly, crediting what providerLaneLiveShape already establishes. A guide that presents working verification as absent is worse than no guide.
  • The guide is net-reducing or states why not. It supersedes or absorbs at least one existing prose passage, and the PR names what it removed. A guide that only adds pages repeats #17147, which was closed as the accretion anti-pattern it was written to fix. → Exercised via the "states why not" branch; the measured basis is the section below.

Why this guide is additive, measured rather than asserted

The AC's first branch was attempted and has no target. Both candidate absorption sources were searched:

  • TenantIngestionModel.md — its one passage on my side of the line is the deletion-telemetry bullet under Cross-Subsystem Surfaces, which states the summary-acceptance contract (array-valued empty errors, attempt-bound receipt, malformed/zero-effect handling, post-ingest checkpoint-write idempotency). Absorbing it would either lose those facts or move ingestion-configuration content into a lane-composition guide — and both documents already declare the opposite split in prose: that guide's ## Related says "This document owns ingestion configuration, triggers and telemetry; that one owns the lane's composition." Absorbing across a boundary two artifacts state explicitly is not a reduction, it is a re-litigation.
  • KnowledgeBase.md — searched for every geometry authority the lane has (parallel, batchEmbeddingChunkSize, safe band, token ceiling, slice budget): zero matches. There is nothing to absorb because the content never existed anywhere, which is the vacuum this ticket was filed about.

Load class, verified not assumed. The accretion defense in AGENTS.md §self_evolving_systems governs loaded bytes. The turn-loaded / skill-loaded surface is enumerated by turn-memory-pre-flight: AGENTS.md, AGENTS_ATLAS.md, .agents/skills/**, .codex/CODEX.md, .claude/CLAUDE.md, .agents/ANTIGRAVITY_RULES.md. learn/** is not in that list — a guide is retrieved on demand by a reader or by KB retrieval, so these lines cost zero per-turn bytes. That is a materially different cost class from a rule, and it is why an additive guide is not the same act as an additive gate.

The decay this mitigates is measured, not projected. Establishing the lane's behaviour cost one full session and produced nine wrong intermediate claims, eight answerable from already-committed content (the table above enumerates them with their sources). That is the recurring cost the artifact removes, and it recurs per investigator rather than once.

Retirement triggers — two, both checkable:

  1. The guide folds into KnowledgeBase.md when the lane acquires a single owning subsystem — i.e. when the four-owner map in section 6 collapses to one. The guide exists because ownership is distributed; if that stops being true, its reason to exist stops with it.
  2. The plane-fact sections shrink to a pointer when the plane facts become machine-derived — when providerLaneLiveShape (or a successor) emits observed concurrency and utilization into the deployment-state snapshot, every dated observation in this guide becomes a query, and those sections retire into it rather than being maintained by hand.

Neither trigger is a date, because a date would expire while the condition still held. Both are conditions a future reader can evaluate against the tree in one command.

  • learn/tree.json registers it; guide lint passes; every relative link resolves.
  • No client identifiers. Plane evidence appears as revision hashes and opaque repo hashes only.
  • Every plane fact carries an observation date AND its re-derivation command. Claims that can change without a repository edit — deployed revision, materialized chunk counts, which reconciliation signal has fired — appear as <value> (observed <date>) beside the command that re-reads them. A plane fact written in the timeless present fails this criterion; a plane fact with no stated value also fails it, because the delta is what makes the guide cheap to maintain.
  • A staleness-check pass exists and is one section. A reader can re-validate every plane fact in the guide by running a listed sequence of commands, and the guide names what to update when a value has moved. Updating a dated guide must be demonstrably cheaper than re-deriving it, which is the whole premise.
  • The merged-versus-deployed question is answerable from the guide alone. A reader can determine, without prior context, whether a given fix is live on a given plane. This session's most consequential finding was a merged-but-undeployed signal, and nothing in the repository made that question routine to ask.

Out of Scope

  • Layer provenance and the retirement list — which of the ~25 coordinating mechanisms is still load-bearing is undecidable until #17412 lands and the lane runs at its declared width. That pass is a follow-up leaf under #17411; this ticket draws the current shape, not the target one.
  • The ADR-0019 challenge. One genuinely uncatalogued pattern was found — deriving a policy the leaf does not express from the leaf's value (slotHeadroomWidth = parallel − 1, where the leaf declares in-flight width and the consumer computes a per-request input count). That is an ADR amendment, filed separately, not smuggled into a guide.
  • Any behaviour change. Documentation only.
  • TenantIngestionModel.md's ingestion-configuration scope, which stays where it is.

Avoided Traps

  • A guide that only adds pages. #17147's precedent is explicit; the net-reduction criterion above is the guard.
  • Presenting the architecture as uniformly broken. It is not. The memory ceiling is measured, the shape verification is real and currently reporting a true finding, and the byte heuristic is deliberately conservative at 3 bytes/token with a stated 1.35× drift factor. A guide that flattens that loses the reader's trust and mislabels the one axis that is actually unverified.
  • Reconstructing behaviour from code reading alone. Several of this session's intermediate claims were plausible readings of the source that the running plane refuted. Where the guide describes behaviour, it cites an observation.
  • Conflating contract and plane. The single highest-value thing this guide can prevent is a correct statement about the code being read as a statement about the deployment — the shape of this session's most expensive finding, a merged-but-undeployed signal. Contract and plane facts are visually separated; plane facts are dated and carry their query.
  • Refusing to state anything that decays. The opposite failure, and one an earlier draft of this ticket committed. A guide that only asks questions forces every reader back to first principles. Stating a dated value is what turns rediscovery into a diff.

Related

Parent: #17411. Sibling: #17412. Outcome bar: D#17136. Documentation practice: D#17326.

tobiu referenced in commit fd10f0f - "docs(agentos): the embedding lane gets an owning document (#17413) on Aug 21, 2026, 2:32 AM
tobiu referenced in commit 4022176 - "docs(agentos): the guide registers itself, and its lessons leave the repository (#17413) on Aug 21, 2026, 2:32 AM
tobiu referenced in commit d30434d - "docs(agentos): the embedding lane gets an owning document (#17413) (#17434) on Aug 21, 2026, 9:15 PM
tobiu closed this issue on Aug 21, 2026, 9:15 PM