Document llama.cpp cloud-deployment profile

Context

This ticket captures the residual docs scope surfaced after the #12090 closeout. #12090 is closed and was resolved by PR #12118, which shipped the generic provider-capacity warning substrate plus docs in learn/agentos/DeploymentCookbook.md and learn/agentos/SharedDeployment.md.

Operator correction on 2026-06-01: the missing work is not to reopen #12090. The missing work is a dedicated llama.cpp cloud-deployment guide or focused guide enhancement inside learn/agentos/cloud-deployment/.

V-B-A Evidence

• Live GitHub state: #12090 is CLOSED, closed on 2026-05-27T23:44:44Z.
• PR #12118 is MERGED, merged on 2026-05-27T23:44:43Z, and closed #12090.
• PR #12118 docs-side files were learn/agentos/DeploymentCookbook.md and learn/agentos/SharedDeployment.md; it did not touch learn/agentos/cloud-deployment/.
• Current repo search shows llama.cpp only appears in learn/agentos/cloud-deployment/Day0Tutorial.md as part of the broad OpenAI-compatible fallback row. There is no dedicated llama-server, /slots, /health, /metrics, or llama.cpp residency-smoke profile in the cloud-deployment guide tree.
• Official llama.cpp server docs describe llama-server as an OpenAI-compatible HTTP server with chat/embeddings routes, context-size/server-slot flags, slots monitoring, health/metrics surfaces, and embeddings mode: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md

The Problem

Neo now has generic local-provider coexistence substrate (requireParallelModels, /v1/models probing, warning-only readiness), but cloud operators do not have a concrete llama.cpp profile that explains how to use that substrate safely.

The risky failure mode is operator-visible and severe: a llama.cpp deployment can appear to work for a single chat call or a single embedding call while still being invalid for Agent OS workloads if chat and embedding roles cannot remain resident together. REM/session-summary/embedding paths alternate between roles; model swapping or context rebuilds defeats the local-provider economics.

The Architectural Reality

• Neo's provider key remains openAiCompatible for llama.cpp unless implementation evidence proves a first-class provider is necessary.
• ai/config.template.mjs already defines openAiCompatible.model, openAiCompatible.embeddingModel, openAiCompatible.keep_alive, and openAiCompatible.requireParallelModels.
• ai/provider/OpenAiCompatible.mjs owns chat-completions transport.
• ai/services/memory-core/TextEmbeddingService.mjs owns OpenAI-compatible /v1/embeddings calls.
• ai/services/graph/ProviderReadinessHelper.mjs uses /v1/models for OpenAI-compatible provider readiness/capacity visibility.
• learn/agentos/cloud-deployment/Day0Tutorial.md distinguishes native Ollama from OpenAI-compatible fallback, but does not yet document a llama.cpp deployment profile.

The Fix

Add a new guide or focused guide enhancement under learn/agentos/cloud-deployment/ for a llama.cpp OpenAI-compatible cloud profile.

The guide should cover:

• llama.cpp as an openAiCompatible backend, not a new Neo provider key by default.
• Supported topology for chat and embedding roles, including whether the safest shape is one llama-server endpoint that exposes both configured model ids or role-separated llama.cpp services.
• If role-separated services are required and current Neo config cannot express separate chat/embedding hosts, state that limitation explicitly and file/follow an implementation ticket rather than hiding it in docs.
• Env examples for NEO_MODEL_PROVIDER=openAiCompatible, NEO_EMBEDDING_PROVIDER=openAiCompatible, NEO_GRAPH_PROVIDER=openAiCompatible, NEO_OPENAI_COMPATIBLE_HOST, NEO_OPENAI_COMPATIBLE_MODEL, NEO_OPENAI_COMPATIBLE_EMBEDDING_MODEL, NEO_OPENAI_COMPATIBLE_KEEP_ALIVE=-1, NEO_OPENAI_COMPATIBLE_REQUIRE_PARALLEL_MODELS=2, and role-keyed NEO_LOCAL_MODELS_* context caps.
• llama.cpp-specific operator smoke: /health, /v1/models, /v1/chat/completions, /v1/embeddings, /slots, and optional metrics when enabled.
• Cross-links from Day0Tutorial.md, Configuration.md, or Overview.md so cloud-deployment readers can find the profile.

Contract Ledger Matrix

Decision Record Impact

Aligned with ADR 0014 cloud-deployment topology and scheduler-task taxonomy. No ADR superseded.

Acceptance Criteria

• A llama.cpp cloud-deployment guide or focused guide enhancement exists under learn/agentos/cloud-deployment/.
• The guide states that llama.cpp uses Neo's existing openAiCompatible provider key initially; adding llamaCpp is out of scope unless separately proven.
• The guide documents the chat+embedding dual-residency invariant and explicitly rejects swapping/rebuild topologies for Agent OS workloads.
• The guide includes an env example for provider selectors, model names, keep-alive, parallel-model requirement, and local role context caps.
• The guide includes llama.cpp-specific smoke checks for /health, /v1/models, /v1/chat/completions, /v1/embeddings, /slots, and metrics when enabled.
• The guide cross-links from the existing cloud-deployment navigation surface (Overview.md, Day0Tutorial.md, or Configuration.md) so operators can find it.
• If the only correct topology requires separate chat and embedding hosts that current Neo config cannot express, the PR documents that as a current limitation and links/files the implementation follow-up.

Out of Scope

• Adding a first-class llamaCpp provider selector.
• Changing provider defaults.
• Retuning embedding context defaults.
• Reworking #12090's already-shipped warning substrate.
• Live benchmarking a production llama.cpp deployment.

Avoided Traps

• Do not comment on or reopen closed #12090 as the active handoff.
• Do not treat generic OpenAI-compatible docs as sufficient for a backend with llama.cpp-specific slot/cache/runtime behavior.
• Do not fossilize unverified llama.cpp CLI flags. Verify against the current official server docs while implementing.
• Do not document a topology that works only by serially swapping chat and embedding models.

Related

• Closed: #12090
• Resolving PR: #12118
• Generic cloud docs: learn/agentos/DeploymentCookbook.md, learn/agentos/SharedDeployment.md
• Target guide tree: learn/agentos/cloud-deployment/
• Official llama.cpp server docs: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md

Handoff Retrieval Hint: "llama.cpp openAiCompatible cloud deployment profile dual resident chat embedding guide #12118 residual"