LearnNewsExamplesServices
Frontmatter
id17012
titleProvider residency repair can warm a 131K chat model into an active heavy-maintenance lane
stateClosed
labels
bugaiarchitectureperformanceagent-os
assigneesneo-gpt
createdAtAug 12, 2026, 12:26 PM
updatedAtAug 12, 2026, 3:01 PM
githubUrlhttps://github.com/neomjs/neo/issues/17012
authorneo-gpt
commentsCount0
parentIssue16706
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtAug 12, 2026, 3:01 PM

Provider residency repair can warm a 131K chat model into an active heavy-maintenance lane

Closed Backlog/active-chunk-15 bugaiarchitectureperformanceagent-os
neo-gpt
neo-gpt commented on Aug 12, 2026, 12:26 PM

Problem

A provider-residency recovery action can warm a large chat model while an orchestrator heavy-maintenance task is already consuming the same constrained local inference capacity.

The live failing receipt is #16706 comment 5265232411:

  • at 09:37:35Z, ProviderReadinessHelper warmed the chat role at num_ctx 131072 through /api/chat during active tenant ingestion;
  • qwen embedding and gemma chat then split a four-core cgroup at roughly two cores each;
  • embedding batches that had completed in about 16 seconds stretched past 24 minutes, and only three provider activities completed from 09:38 through 09:55.

The orchestrator already serializes its heavy-maintenance tasks. Its log can truthfully say that tenant sync is deferred while session summarization is active. But warm-provider is not enrolled in that same contention boundary, so the recovery loop can create the concurrency the scheduler prevented.

This is a pre-dispatch admission defect. The post-timeout / abandoned-provider-work tail remains #16853 and is not part of this ticket.

Verify-before-assert receipt

The current source was inspected before choosing the remedy:

  • MaintenanceBackpressureService.getActiveHeavyMaintenanceTask() and the shared heavy-maintenance lease already expose the active resource-mutex lane.
  • ContainerHealthControllerService.actuate() carries isEffectStillAdmitted only for ollama-residual-load-restart.
  • RecoveryActuatorService.executeTargetAction() forwards that oracle only into the compose restart path; warm-provider drops it.
  • repairProviderRoleSetResidency() and ensureOllamaModelsReady() already carry the authority oracle through awaited probes and re-check it immediately before each role warm. The missing demand oracle belongs beside that proven shape.
  • warmOllamaRoleModel() intentionally uses the configured role context. Warming a smaller context would leave the role below its readiness requirement and invite another repair attempt.
  • PR #17007's admission queue is ask-specific and process-local. It cannot govern an orchestrator warm against Memory Core / Knowledge Base embeddings.

A live issue search, the newest 20 open-ticket sweep, the recent A2A queue, local ticket mirrors, and a Memory Core raw-memory sweep found no owner for this exact pre-warm admission seam. The semantic memory query was a miss; the public receipt and current source are the authority.

Intended solution

Extend the existing last-boundary effect-admission path; do not add a second scheduler, a new config leaf, or a provider-specific global queue.

  1. The orchestrator derives a fail-closed warm-provider admission decision from the existing heavy-maintenance task state and shared lease.
  2. The controller attaches the existing isEffectStillAdmitted oracle to warm-provider decisions as well as residual restarts.
  3. The actuator carries the oracle through warmProviderResidency() and repairProviderRoleSetResidency().
  4. Native Ollama readiness re-checks the oracle after its read-only probe and immediately before each /api/chat or /api/embed warm, while retaining the disposition of earlier role warms.
  5. A denial before the first provider mutation uses runtime-effect-not-admitted and becomes {status: 'declined', reasonCode: 'effect-no-longer-admitted'}. It must not charge an anti-thrash attempt for provider work that never started.
  6. A denial after an earlier role warm has already left Neo refuses the next warm but is not a zero-effect decline. It carries an explicit partial-effect disposition into the audited recovery terminal and charges one bounded attempt; otherwise the first warm would disappear from the recovery truth.
  7. Callers that do not supply the oracle keep their current startup/readiness behavior byte-for-byte.

This is a conservative yield, not preemption. The active heavy lane finishes; a later observation can retry residency repair after the lease clears.

Acceptance criteria

  • With a running heavy-maintenance task, a warm-provider controller decision reaches a declined terminal and dispatches no provider warm.
  • With no running task but an active shared heavy-maintenance lease, the same action fails closed and dispatches no provider warm.
  • If heavy demand begins after the initial readiness probe but before the first role warm, that warm is refused and no provider mutation is dispatched.
  • If heavy demand begins between two role warms, the next native Ollama warm is refused while the earlier warm remains visible as a partial provider effect.
  • A pre-first-warm refusal preserves runtime-effect-not-admitted through the helper and actuator, writes no action receipt, and consumes no anti-thrash attempt.
  • A post-first-warm refusal writes no successful action receipt, but it does write a truthful partial/failed recovery terminal and consumes one bounded anti-thrash attempt.
  • With no active heavy demand, the existing role-set repair still warms the configured model at the configured context; no context reduction is introduced.
  • Startup/readiness callers that omit isEffectStillAdmitted retain current behavior.
  • Heavy-maintenance inspection failure is fail-closed and observable in the orchestrator log.
  • ADR-0026 §2.4/§2.5 is amended in the same PR to state that warm-provider is admitted only while the live heavy-maintenance demand boundary remains clear; ADR-0022 remains the scheduling/fairness authority.
  • The composition proof drives controller → actuator → provider helper, rather than asserting only a detached predicate.

Required red/green evidence

A deterministic fixture must begin with the current red behavior:

  1. mark tenant-repo-sync (or an equivalent registered heavy task) running;
  2. emit a provider-role-residency / warm-provider decision;
  3. observe that the current implementation calls the warm seam.

After the fix, the same fixture must prove zero warm calls and the exact declined terminal. A second fixture flips the oracle before the first warm after the readiness probe and proves the same zero-effect result. A third fixture flips the oracle after role 1 has warmed: role 2 must not dispatch, while the actuator records an explicit partial-effect failure rather than claiming nothing happened. A positive control keeps the oracle admitted and proves the existing warm payload, including num_ctx, is unchanged.

Contract Ledger

Producer Consumer Surface / Type Fallback / failure Evidence
Orchestrator ContainerHealthControllerService existing isEffectStillAdmitted(decision) -> Boolean, extended to warm-provider active task, active lease, or inspection uncertainty => false; omitted oracle preserves current behavior orchestration composition spec
ContainerHealthControllerService RecoveryActuatorService.apply() existing optional isEffectStillAdmitted: () => Boolean attached for warm-provider; absent for unrelated actions controller spec with strict call envelope
RecoveryActuatorService repairProviderRoleSetResidency() / ensureOllamaModelsReady() the same optional live oracle, carried through awaited boundaries plus the earlier-role effect disposition denial before the first warm throws reason='runtime-effect-not-admitted'; denial after an earlier warm throws reason='runtime-effect-partially-applied' with bounded provider-result facts; null means no demand gate actuator + provider-readiness composition spec
RecoveryActuatorService.apply() heal/recovery observers existing declined terminal for zero-effect denial; existing failed terminal extended with effectDisposition='partial' for post-first-warm denial zero effect => no attempt/no receipt; partial effect => one bounded attempt + failure audit, never a success receipt recovery-run / attempt-store positive and negative assertions

No new external API, OpenAPI field, config leaf, or public method is introduced.

Decision record impact

  • ADR-0026: amended in the implementation PR. This ticket narrows when the already-admitted warm-provider action may leave the process; it does not add an action or widen privilege.
  • ADR-0022: aligned. Heavy-maintenance remains a single resource mutex and remains non-preemptive; residency repair yields to that declared lane.
  • ADR-0019: no config change is intended.

Avoided traps

  • Do not warm with a smaller num_ctx. Readiness validates the configured context; a smaller warm preserves the insufficiency and can create a repair loop.
  • Do not reuse the ask queue as global provider authority. It is ask-specific and process-local.
  • Do not fold provider cancellation / stranded-runner recovery. #16853 owns that distinct post-dispatch concern.
  • Do not preempt or cancel the active heavy task. This ticket prevents a new warm from joining it.
  • Do not name a deployment or customer in public artifacts. The receipt is described only as a constrained external plane.

Scope

One Neo-side PR targeting dev: orchestrator/controller/actuator/provider-readiness plumbing, focused specs, and the ADR-0026 amendment. No external-plane experiment or deployment mutation is required.

Origin Session ID: 019fe0b1-114b-7c30-aaf4-8317c1f99d4b

Memory Core prior-art sweep: explicit semantic miss for ProviderReadinessHelper chat warm during tenant ingestion; live source + the failing receipt govern this ticket.

tobiu referenced in commit 5d14d83 - "feat(orchestrator): gate provider warms on heavy demand (#17012) (#17016)" on Aug 12, 2026, 3:01 PM
tobiu closed this issue on Aug 12, 2026, 3:01 PM