The provider-lane healthcheck sha256-sums the whole model file every 15s, so a busy engine is marked unhealthy and restarted mid-inference

Context

ai/deploy/docker-compose.provider-lanes.yml:215 ships this steady-state healthcheck for the embedding lane:

test: ["CMD-SHELL", "curl --fail --silent http://127.0.0.1:8080/health >/dev/null && echo \"$NEO_PROVIDER_LANE_MODEL_SHA256  /models/Qwen3-Embedding-8B-Q4_K_M.gguf\" | sha256sum -c - >/dev/null"]
interval: 15s
timeout : 10s

The same shape appears at line 171 for the Ollama lane (manifest hash) and the download-verify at 188.

The probe hashes a 4.7 GB model file every 15 seconds against a 10-second timeout, on the same CPU allocation the inference engine is using.

The Problem

Verifying the weights after download is correct — that is what the start_period and the entrypoint check at line 188 are for. Leaving the hash in the recurring probe means the liveness signal is dominated by an I/O + CPU operation that has nothing to do with liveness, and that competes directly with the workload it is supposed to observe.

Measured on an external plane, 2026-08-13:

• probeReliability: sampleCount 5, failureCount 5, failureRate 1.0 — the probe never passes once the engine is under load.
• The container is consequently marked unhealthy and restarted by the container-health controller. Three separate llama-server incarnations are visible in a single 120-line log tail, each re-loading the 8B model (~9.5 s per load_model → model loaded).
• Every restart destroys in-flight embedding work. Requests that were mid-flight surface to callers as KB_VECTOR_EMBED_CONNECTION_REFUSED.

The failure is self-reinforcing: load makes the hash slower, the slower hash fails the probe, the failed probe restarts the engine, the restart discards work, the discarded work is retried, and the retry adds load.

A liveness probe must be cheap and must not compete with the thing it measures. This one is O(model size) and runs four times a minute forever.

Architectural Reality

• curl /health alone is a complete liveness signal for llama-server; it answers only when the model is loaded and the server is serving.
• Weight integrity is a boot-time property. It is already verified by the entrypoint before exec, and a file that passed at boot does not silently change under a running container.
• If periodic integrity verification is genuinely wanted, it belongs on a slow independent cadence with its own reporting, not inside the probe that gates restarts.

The Fix (shape)

1. Remove the hash from the recurring healthcheck test; keep curl --fail /health.
2. Keep integrity verification at the entrypoint (already present) — that is the boundary where a corrupt or partial download must be caught.
3. If a periodic integrity check is wanted, give it its own cadence (hours, not seconds) and a reporting surface that does not feed the restart decision.
4. Audit the sibling lanes at lines 171 and 215 together; the Ollama manifest hash is smaller but the same class.

Acceptance Criteria

• The provider-lane healthcheck contains no sha256sum over model weights or manifests.
• Boot-time integrity verification is retained and demonstrably still rejects a corrupt/partial model file.
• A fixture holds the engine at sustained high CPU and asserts the healthcheck still passes within its timeout.
• If a periodic integrity check is added, it cannot mark the container unhealthy on its own.

Out of Scope

• Engine batch/context tuning.
• Host resource allocation — this reproduces at any allocation, because the hash cost scales with model size and the probe interval is fixed.

Avoided Traps

• "Raise the timeout." Hides it until the model or the load grows. The probe should not be doing this work at all.
• "Give the container more CPU." The probe would still burn a core hashing gigabytes on a fixed cadence, in competition with inference.
• "It only matters under load." Under load is exactly when a liveness probe must be trustworthy.

Related

• #17062 — embedding admission ordering on the same plane
• #17048 — engine-slot monopoly
• #16853 — stranded provider work after client disconnect