A container's startup facts age out of the log tail before anyone reads them

Context

Measured friction from the same 2026-08-18 incident as #17356. An embedding container was at 96.7% of a 32 GB ceiling. I had a formula from a prior session predicting the peak — peak(GiB) ≈ 7.69 + 8.42e-8 × n² — which gives 30.29 GiB at a full batch. Observed: 30.95 GiB, within 2.2%.

So the prediction held, but I could not attribute it. Of 30.95 GiB, roughly 7.6 was accounted for by measured KV plus weights; the remaining ~23 was attributed to a quadratic attention buffer by inference from a formula, not by reading the engine's own numbers.

Those numbers existed. llama.cpp prints its KV-cache size, compute-buffer sizes and graph splits at startup, in the same log stream the snapshot already carries. The container had been up four hours, logs.tail is 120 lines, and the banner was thousands of lines gone.

The diagnosis therefore rested on a fitted model where a direct measurement had been available and was discarded by retention.

The Problem

services[].logs is {appliedSince, appliedUntil, incarnationBounded, maxBytes, tail, text, truncated} — a rolling window sized for recent activity, which is the right default for "what is this service doing now". It is structurally the wrong instrument for "what did this service decide when it started", because those facts are emitted exactly once, at the moment least likely to coincide with anyone looking.

The class is broader than one engine's banner. Startup emissions are the place a process reports its resolved geometry, allocation plan, model identity and feature negotiation — and every one of them is unreadable by the time a human notices a problem. incarnationBounded already acknowledges that an incarnation is the meaningful unit; the retention window does not follow it.

Raising logTail is not the fix. The banner's distance from the tail grows with uptime, so any fixed window is a bet on how soon someone looks, and the bet gets worse the longer a deployment runs well.

The Architectural Reality

The bridge already observes services on a cadence and already computes durable records that outlive a window — restartChurn keeps a baseline across restarts precisely because inspect.state describes only the live incarnation, and memoryPressure.receipt retains a sampling window's evidence rather than an instant.

So the pattern for "a fact that must survive its observation window" exists here twice. Startup output is a third instance and currently the only one still window-bound.

incarnationBounded is the natural key: a container's startup facts are valid for exactly one incarnation, and a restart invalidates them rather than appending to them.

The Fix

Capture a bounded head of each incarnation's log stream once, and carry it beside the tail.

• On first observation of a new incarnation — detected by the same signal restartChurn already tracks — retain a bounded prefix of that incarnation's output.
• Store it as logs.startup: {text, capturedAt, lines, truncated, incarnationStartedAt}, distinct from logs.text so no consumer reading the tail changes behaviour.
• A restart replaces it wholesale. It is never appended to and never merged across incarnations, or it stops answering the question it exists for.
• Bounded by the same logMaxBytes discipline as the tail, configured beside it.

The startup head is inherently small: it is emitted in the first seconds and then stops, so a modest cap captures all of it rather than a sample.

Contract Ledger Matrix

The fallback row is the one that matters. A bridge that starts after a container will never have that container's startup output, and the projection must say so. An empty string that reads as "this service printed nothing at startup" is the failure mode to design against.

Decision Record impact

none — an additive projection field on an existing record, following the durability pattern restartChurn established.

Acceptance Criteria

• AC-1: for a service observed from its start, logs.startup carries that incarnation's opening output while logs.text continues to carry the recent tail; both present, neither altered by the other.
• AC-2: a restart replaces logs.startup and updates incarnationStartedAt. A spec asserts the previous incarnation's text is gone — appending is the defect, because two incarnations' startup facts read as one and the older values look current.
• AC-3 (the honesty AC): when the bridge first observes a service already running, logs.startup is absent with an explicit reason. A spec asserts it is not an empty string and not a truncated tail masquerading as a head.
• AC-4: the retained head is bounded, and a service that logs continuously from startup does not grow the snapshot without limit.
• AC-5 (the motivating case): the memory attribution that prompted this is answerable from the snapshot alone — an embedding provider's reported KV-cache and compute-buffer sizes readable after hours of uptime, with no container restart required to recover them.

Out of Scope

• Raising logTail or logMaxBytes. A larger window is a longer bet on the same wrong instrument; the banner's distance from the tail grows with uptime.
• Full log retention or shipping. This is a bounded head for diagnosis, not a log pipeline.
• Parsing engine-specific startup formats. The bridge retains text; interpreting a llama.cpp banner is a reader's concern and would couple the bridge to one provider.
• Resolved configuration — #17356 covers that, and it is declared config rather than observed startup output. The two are complementary: one says what we asked for, the other what the process decided.

Avoided Traps

• Treating this as a log-retention problem. It is a record-durability problem. Framing it as retention leads to a bigger buffer; framing it as durability leads to a once-captured field, which is what restartChurn already does for a neighbouring fact.
• Merging incarnations. Appending would make the field grow monotonically and would surface a dead incarnation's geometry as current — worse than absence, because it reads as authoritative.
• Shipping AC-1 without AC-3. The common case is a bridge that started first, so the absent case is rare in testing and normal in production. An empty string there produces a confident wrong reading, which is how this class of gap caused the fitted-model diagnosis in the first place.

Related

• #17356 — resolved-config projection; the declared-input half of the same blind spot.
• #17349 / #17345 — tenant-sync behaviours diagnosed during the same incident.

Handoff Retrieval Hints

• Retrieval Hint: "startup banner aged out of log tail, buffer accounting unreadable"
• Retrieval Hint: "capture incarnation head once, restartChurn durability pattern"
• The pattern to copy: restartChurn's baseline, which survives a window for the same reason.
• Live latest-open sweep: checked latest 8 open issues (recorded on #17356, same turn) plus keyword sweeps for log tail startup banner, container startup facts snapshot, provider buffer accounting, logs incarnation bounded — no equivalent found.

Origin Session ID: 9ccc2fa1-8843-4796-8e85-5e151c0392d2

— Vega (Claude Opus 5, Claude Code) 🌿