Problem
A provider-residency recovery action can warm a large chat model while an orchestrator heavy-maintenance task is already consuming the same constrained local inference capacity.
The live failing receipt is #16706 comment 5265232411:
- at 09:37:35Z,
ProviderReadinessHelper warmed the chat role at num_ctx 131072 through /api/chat during active tenant ingestion;
- qwen embedding and gemma chat then split a four-core cgroup at roughly two cores each;
- embedding batches that had completed in about 16 seconds stretched past 24 minutes, and only three provider activities completed from 09:38 through 09:55.
The orchestrator already serializes its heavy-maintenance tasks. Its log can truthfully say that tenant sync is deferred while session summarization is active. But warm-provider is not enrolled in that same contention boundary, so the recovery loop can create the concurrency the scheduler prevented.
This is a pre-dispatch admission defect. The post-timeout / abandoned-provider-work tail remains #16853 and is not part of this ticket.
Verify-before-assert receipt
The current source was inspected before choosing the remedy:
MaintenanceBackpressureService.getActiveHeavyMaintenanceTask() and the shared heavy-maintenance lease already expose the active resource-mutex lane.
ContainerHealthControllerService.actuate() carries isEffectStillAdmitted only for ollama-residual-load-restart.
RecoveryActuatorService.executeTargetAction() forwards that oracle only into the compose restart path; warm-provider drops it.
repairProviderRoleSetResidency() and ensureOllamaModelsReady() already carry the authority oracle through awaited probes and re-check it immediately before each role warm. The missing demand oracle belongs beside that proven shape.
warmOllamaRoleModel() intentionally uses the configured role context. Warming a smaller context would leave the role below its readiness requirement and invite another repair attempt.
- PR #17007's admission queue is ask-specific and process-local. It cannot govern an orchestrator warm against Memory Core / Knowledge Base embeddings.
A live issue search, the newest 20 open-ticket sweep, the recent A2A queue, local ticket mirrors, and a Memory Core raw-memory sweep found no owner for this exact pre-warm admission seam. The semantic memory query was a miss; the public receipt and current source are the authority.
Intended solution
Extend the existing last-boundary effect-admission path; do not add a second scheduler, a new config leaf, or a provider-specific global queue.
- The orchestrator derives a fail-closed
warm-provider admission decision from the existing heavy-maintenance task state and shared lease.
- The controller attaches the existing
isEffectStillAdmitted oracle to warm-provider decisions as well as residual restarts.
- The actuator carries the oracle through
warmProviderResidency() and repairProviderRoleSetResidency().
- Native Ollama readiness re-checks the oracle after its read-only probe and immediately before each
/api/chat or /api/embed warm, while retaining the disposition of earlier role warms.
- A denial before the first provider mutation uses
runtime-effect-not-admitted and becomes {status: 'declined', reasonCode: 'effect-no-longer-admitted'}. It must not charge an anti-thrash attempt for provider work that never started.
- A denial after an earlier role warm has already left Neo refuses the next warm but is not a zero-effect decline. It carries an explicit partial-effect disposition into the audited recovery terminal and charges one bounded attempt; otherwise the first warm would disappear from the recovery truth.
- Callers that do not supply the oracle keep their current startup/readiness behavior byte-for-byte.
This is a conservative yield, not preemption. The active heavy lane finishes; a later observation can retry residency repair after the lease clears.
Acceptance criteria
Required red/green evidence
A deterministic fixture must begin with the current red behavior:
- mark
tenant-repo-sync (or an equivalent registered heavy task) running;
- emit a
provider-role-residency / warm-provider decision;
- observe that the current implementation calls the warm seam.
After the fix, the same fixture must prove zero warm calls and the exact declined terminal. A second fixture flips the oracle before the first warm after the readiness probe and proves the same zero-effect result. A third fixture flips the oracle after role 1 has warmed: role 2 must not dispatch, while the actuator records an explicit partial-effect failure rather than claiming nothing happened. A positive control keeps the oracle admitted and proves the existing warm payload, including num_ctx, is unchanged.
Contract Ledger
| Producer |
Consumer |
Surface / Type |
Fallback / failure |
Evidence |
Orchestrator |
ContainerHealthControllerService |
existing isEffectStillAdmitted(decision) -> Boolean, extended to warm-provider |
active task, active lease, or inspection uncertainty => false; omitted oracle preserves current behavior |
orchestration composition spec |
ContainerHealthControllerService |
RecoveryActuatorService.apply() |
existing optional isEffectStillAdmitted: () => Boolean |
attached for warm-provider; absent for unrelated actions |
controller spec with strict call envelope |
RecoveryActuatorService |
repairProviderRoleSetResidency() / ensureOllamaModelsReady() |
the same optional live oracle, carried through awaited boundaries plus the earlier-role effect disposition |
denial before the first warm throws reason='runtime-effect-not-admitted'; denial after an earlier warm throws reason='runtime-effect-partially-applied' with bounded provider-result facts; null means no demand gate |
actuator + provider-readiness composition spec |
RecoveryActuatorService.apply() |
heal/recovery observers |
existing declined terminal for zero-effect denial; existing failed terminal extended with effectDisposition='partial' for post-first-warm denial |
zero effect => no attempt/no receipt; partial effect => one bounded attempt + failure audit, never a success receipt |
recovery-run / attempt-store positive and negative assertions |
No new external API, OpenAPI field, config leaf, or public method is introduced.
Decision record impact
- ADR-0026: amended in the implementation PR. This ticket narrows when the already-admitted
warm-provider action may leave the process; it does not add an action or widen privilege.
- ADR-0022: aligned. Heavy-maintenance remains a single resource mutex and remains non-preemptive; residency repair yields to that declared lane.
- ADR-0019: no config change is intended.
Avoided traps
- Do not warm with a smaller
num_ctx. Readiness validates the configured context; a smaller warm preserves the insufficiency and can create a repair loop.
- Do not reuse the ask queue as global provider authority. It is ask-specific and process-local.
- Do not fold provider cancellation / stranded-runner recovery. #16853 owns that distinct post-dispatch concern.
- Do not preempt or cancel the active heavy task. This ticket prevents a new warm from joining it.
- Do not name a deployment or customer in public artifacts. The receipt is described only as a constrained external plane.
Scope
One Neo-side PR targeting dev: orchestrator/controller/actuator/provider-readiness plumbing, focused specs, and the ADR-0026 amendment. No external-plane experiment or deployment mutation is required.
Origin Session ID: 019fe0b1-114b-7c30-aaf4-8317c1f99d4b
Memory Core prior-art sweep: explicit semantic miss for ProviderReadinessHelper chat warm during tenant ingestion; live source + the failing receipt govern this ticket.
Problem
A provider-residency recovery action can warm a large chat model while an orchestrator heavy-maintenance task is already consuming the same constrained local inference capacity.
The live failing receipt is #16706 comment 5265232411:
ProviderReadinessHelperwarmed the chat role atnum_ctx 131072through/api/chatduring active tenant ingestion;The orchestrator already serializes its heavy-maintenance tasks. Its log can truthfully say that tenant sync is deferred while session summarization is active. But
warm-provideris not enrolled in that same contention boundary, so the recovery loop can create the concurrency the scheduler prevented.This is a pre-dispatch admission defect. The post-timeout / abandoned-provider-work tail remains #16853 and is not part of this ticket.
Verify-before-assert receipt
The current source was inspected before choosing the remedy:
MaintenanceBackpressureService.getActiveHeavyMaintenanceTask()and the shared heavy-maintenance lease already expose the active resource-mutex lane.ContainerHealthControllerService.actuate()carriesisEffectStillAdmittedonly forollama-residual-load-restart.RecoveryActuatorService.executeTargetAction()forwards that oracle only into the compose restart path;warm-providerdrops it.repairProviderRoleSetResidency()andensureOllamaModelsReady()already carry the authority oracle through awaited probes and re-check it immediately before each role warm. The missing demand oracle belongs beside that proven shape.warmOllamaRoleModel()intentionally uses the configured role context. Warming a smaller context would leave the role below its readiness requirement and invite another repair attempt.A live issue search, the newest 20 open-ticket sweep, the recent A2A queue, local ticket mirrors, and a Memory Core raw-memory sweep found no owner for this exact pre-warm admission seam. The semantic memory query was a miss; the public receipt and current source are the authority.
Intended solution
Extend the existing last-boundary effect-admission path; do not add a second scheduler, a new config leaf, or a provider-specific global queue.
warm-provideradmission decision from the existing heavy-maintenance task state and shared lease.isEffectStillAdmittedoracle towarm-providerdecisions as well as residual restarts.warmProviderResidency()andrepairProviderRoleSetResidency()./api/chator/api/embedwarm, while retaining the disposition of earlier role warms.runtime-effect-not-admittedand becomes{status: 'declined', reasonCode: 'effect-no-longer-admitted'}. It must not charge an anti-thrash attempt for provider work that never started.This is a conservative yield, not preemption. The active heavy lane finishes; a later observation can retry residency repair after the lease clears.
Acceptance criteria
warm-providercontroller decision reaches a declined terminal and dispatches no provider warm.runtime-effect-not-admittedthrough the helper and actuator, writes no action receipt, and consumes no anti-thrash attempt.isEffectStillAdmittedretain current behavior.warm-provideris admitted only while the live heavy-maintenance demand boundary remains clear; ADR-0022 remains the scheduling/fairness authority.Required red/green evidence
A deterministic fixture must begin with the current red behavior:
tenant-repo-sync(or an equivalent registered heavy task) running;provider-role-residency/warm-providerdecision;After the fix, the same fixture must prove zero warm calls and the exact declined terminal. A second fixture flips the oracle before the first warm after the readiness probe and proves the same zero-effect result. A third fixture flips the oracle after role 1 has warmed: role 2 must not dispatch, while the actuator records an explicit partial-effect failure rather than claiming nothing happened. A positive control keeps the oracle admitted and proves the existing warm payload, including
num_ctx, is unchanged.Contract Ledger
OrchestratorContainerHealthControllerServiceisEffectStillAdmitted(decision) -> Boolean, extended towarm-providerfalse; omitted oracle preserves current behaviorContainerHealthControllerServiceRecoveryActuatorService.apply()isEffectStillAdmitted: () => Booleanwarm-provider; absent for unrelated actionsRecoveryActuatorServicerepairProviderRoleSetResidency()/ensureOllamaModelsReady()reason='runtime-effect-not-admitted'; denial after an earlier warm throwsreason='runtime-effect-partially-applied'with bounded provider-result facts; null means no demand gateRecoveryActuatorService.apply()effectDisposition='partial'for post-first-warm denialNo new external API, OpenAPI field, config leaf, or public method is introduced.
Decision record impact
warm-provideraction may leave the process; it does not add an action or widen privilege.Avoided traps
num_ctx. Readiness validates the configured context; a smaller warm preserves the insufficiency and can create a repair loop.Scope
One Neo-side PR targeting
dev: orchestrator/controller/actuator/provider-readiness plumbing, focused specs, and the ADR-0026 amendment. No external-plane experiment or deployment mutation is required.Origin Session ID:
019fe0b1-114b-7c30-aaf4-8317c1f99d4bMemory Core prior-art sweep: explicit semantic miss for
ProviderReadinessHelper chat warm during tenant ingestion; live source + the failing receipt govern this ticket.