Provider multi-model coexistence mandate — Ollama + OpenAiCompatible defaults ensure chat + embedding stay resident

Problem

Neo's REM cycle alternates between chat model (SemanticGraphExtractor.executeTriVectorExtraction, TopologyInferenceEngine.extractTopology, GoldenPathSynthesizer.synthesizeGoldenPath) and embedding model (memory ingestion, per-session new-memory vectorization). Per session: chat → embedding → chat. 2 switches × ~10 sessions per REM cycle = ~20 switches.

If the underlying provider holds only ONE model resident:

• Every switch evicts the resident model + reloads the other from disk
• ~18s cold-prefill cost per switch (per neo PR #12076 benchmark anchor on local LM Studio gemma-4-31b-it)
• ~6 minutes pure model-swap overhead per REM cycle regardless of actual inference time
• Defeats keep_alive=-1 (neomjs/neo#12089) entirely — the OTHER model's load triggers eviction of the keep_alive-pinned model

Operator framing 2026-05-27 ~09:00Z:

"our own default for our used providers => we MUST always use TWO model support (embedding and chat model coexistence)"

Why this is Neo-level, not consumer-level

Operator-side compose-yml fixes (e.g., OLLAMA_MAX_LOADED_MODELS=2) are workarounds for a Neo-level mandate. The substrate-correct shape: Neo's providers DECLARE the multi-model requirement + BOOT-TIME PROBE the server-side capacity + WARN if insufficient. Operator-side env vars satisfy the requirement; Neo enforces it has been satisfied.

Fix shape (both provider families — symmetry mandate)

Ollama provider

Neo can't directly control OLLAMA_MAX_LOADED_MODELS (server-process startup config, not API-mutable), but can:

1. Declare expected capacity via new config field ollama.requireParallelModels (default 2 for chat+embedding; can grow if operator deploys additional models)
2. Probe at boot via GET <host>/api/ps to verify the running Ollama instance can hold N models OR currently has both pre-warmed
3. WARN at boot when capacity check fails — operator-actionable log line with the specific env var to set

OpenAiCompatible provider

Same shape — LM Studio, llama.cpp, vLLM each have different mechanisms:

• LM Studio (newer versions): "JIT load" or "auto-eviction" UI settings; OpenAI-compat endpoint addresses each via model parameter
• llama.cpp: typically one-model-per-server; multi-model requires separate server instances
• vLLM: multi-model API with per-model state

Neo's provider can't universally enforce, but can:

1. Declare expected capacity via openAiCompatible.requireParallelModels (default 2)
2. Probe at boot via GET <host>/v1/models + cross-reference against expected chat + embedding model names
3. WARN at boot when probe shows only one model accessible OR when behavior probe (small chat call followed by small embedding call within keep_alive window) shows cold-reload cost on the second call

Contract Ledger

Acceptance Criteria

• AC1: NEW config field ollama.requireParallelModels (default 2) in ai/config.template.mjs; env-overridable via NEO_OLLAMA_REQUIRE_PARALLEL_MODELS
• AC2: NEW config field openAiCompatible.requireParallelModels (default 2) — symmetric with Ollama
• AC3: Boot-time capacity probe for Ollama via GET /api/ps; WARN log line [provider/ollama] expected N+ models loaded (chat=X, embedding=Y); observed N loaded — model swap penalty likely; set OLLAMA_MAX_LOADED_MODELS=N+ in ollama server env
• AC4: Boot-time capacity probe for OpenAiCompatible via GET /v1/models; analogous WARN log line
• AC5: Unit tests for both probes (mocked HTTP responses for present/absent model states)
• AC6: Documentation update at learn/agentos/ operator-cookbook describing the multi-model mandate + per-provider env var matrix + verification commands
• AC7: Migration note in PR body — existing deployments that DON'T set the underlying provider's multi-model env var will see boot-time WARN log + may experience model-swap penalty until corrected

Avoided Traps

• ❌ Don't attempt to mutate the underlying server's loaded-models cap via API — Ollama's OLLAMA_MAX_LOADED_MODELS is a process-startup setting, not API-mutable. Neo declares the requirement; operator satisfies it via server-side config.
• ❌ Don't fail boot on capacity probe failure — boot-blocking would regress existing deployments. WARN + observable signal is sufficient; operators act on the WARN.
• ❌ Don't bundle with #12089 — that ticket is keep_alive default substrate; this ticket is multi-model coexistence substrate. Two distinct mandates. Operator-explicit separation.
• ❌ Don't enforce the capacity probe at every callsite — boot-time check is sufficient signal; per-call check would add latency without operational value.

Related

• Epic #12065 (Orchestrator-as-SSOT REM pipeline) — multi-model coexistence prevents the ~6min/cycle model-swap overhead class
• neomjs/neo#12089 — keep_alive=-1 default switch (companion mandate; without multi-model coexistence, keep_alive is defeated by OTHER model's load)
• neomjs/neo#12072 / PR #12084 — Sub 6 Part A provider-readiness substrate (natural home for the capacity probe helper)
• Operator-side docker-mcp env var fix consumed by this Neo-side mandate (parallel deployment-side work)
• PR #12076 benchmark — 18s cold-prefill empirical anchor

Operator anchor

"our own default for our used providers => we MUST always use TWO model support (embedding and chat model coexistence)" — @tobiu 2026-05-27 ~09:00Z