LearnNewsExamplesServices
Frontmatter
id15694
titleMake embedding health and freeze-recovery probes transport-abortable
stateClosed
labels
enhancementaitestingarchitectureperformance
assigneesneo-gpt-emmy
createdAtJul 22, 2026, 11:45 AM
updatedAtJul 23, 2026, 12:32 AM
githubUrlhttps://github.com/neomjs/neo/issues/15694
authorneo-gpt-emmy
commentsCount3
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtJul 23, 2026, 12:32 AM

Make embedding health and freeze-recovery probes transport-abortable

Closed Backlog/active-chunk-8 enhancementaitestingarchitectureperformance
neo-gpt-emmy
neo-gpt-emmy commented on Jul 22, 2026, 11:45 AM

Context

A timeout wrapper can return while the underlying embedding request keeps consuming a shared provider. The current withTimeout() helper is a plain Promise.race(), so the cached Memory Core embedding-write canary abandons its await without cancelling transport. The orchestrator freeze re-probe calls TextEmbeddingService.embedTexts() with neither a signal nor a request deadline.

Two existing consumers need transport-level cancellation:

  1. the regular cached self-diagnostic canary in HealthService.buildEmbeddingWriteCanaryBlock();
  2. the self-healing freeze re-probe in Orchestrator.probeFrozenCollectionHealth(), invoked only when freezeReprobeDecision says a frozen record is due.

There is no separate action-admission canary in scope.

Problem

An abandoned probe is not bounded. It can remain in the OpenAI-compatible priority queue, keep a socket active, continue through retry delays or batch yields, or reject after the diagnostic/recovery decision has already moved on. Under shared-provider load, that stale work can delay the next legitimate embedding or recovery batch.

A generic “provider supports cancellation” claim is also false: the three embedding branches have different local guarantees and different residuals.

Authority Boundary

  • ADR-0025: diagnostics produce evidence; an abort/timeout never selects or executes a heal action.
  • ADR-0027: freeze re-probe may only lift containment through the existing decider and recovery path; inconclusive evidence stays frozen.
  • ADR-0019: each deadline is read from its owning config provider at the use site.
  • Preserved-vector restore is provider-free. This ticket cannot add a restore probe or re-embedding authority.

Decision Record impact: aligned-with ADR-0019, ADR-0025, and ADR-0027; no amendment or supersession.

Consumed API Contract

Extend the existing signatures compatibly:

  • embedText(text, explicitProvider, options = {})
  • embedTexts(texts, explicitProvider, options = {})

The second positional argument remains the required provider selector. The third argument accepts only:

Key Contract
signal Optional upstream AbortSignal; abort must stop all locally-owned queued, delayed, or in-flight work reachable from this invocation.
operationLabel Bounded diagnostic label used in locally-created errors/logs; it never changes provider selection or scheduling priority.

Error semantics are provider-specific where the live providers already differ:

  • Upstream cancellation is never rewritten as a provider-owned timeout.
  • OpenAI-compatible and Ollama paths preserve an Error-valued upstream signal.reason or surface a native/Neo-created AbortError with code='ABORT_ERR' when no Error reason exists.
  • The Gemini SDK wraps fetch cancellation as GoogleGenerativeAIAbortError and drops the caller reason. When the caller signal is aborted, TextEmbeddingService must catch that wrapper and rethrow the exact Error-valued signal.reason; if no Error-valued reason exists, it must throw a bounded Neo-created AbortError with code='ABORT_ERR' and operationLabel. A Gemini SDK error observed while the caller signal is not aborted remains the original SDK error.
  • Existing OpenAI-compatible provider deadlines remain code='OPENAI_COMPATIBLE_REQUEST_TIMEOUT'; the existing contention classifier and its tests retain that compatibility contract.
  • Existing Ollama provider deadlines remain code='PROVIDER_TIMEOUT'.
  • A consumer-owned probe deadline aborts with an Error-valued reason carrying code='EMBEDDING_PROBE_TIMEOUT', operationLabel, and timeoutMs; the adapter preserves that object and the consumer maps it to inconclusive diagnostic evidence.
  • Every Neo-owned timer, abort listener, and queue entry is removed or made unreachable exactly once when the operation settles. The installed Gemini SDK's internal controller/listener lifetime is an admitted library residual; Neo does not claim to remove SDK-owned listeners.

Provider Capability Matrix

Provider branch Required local behavior Honest residual
OpenAI-compatible Reject an already-aborted call before provider preflight or enqueue. Remove an aborted queued item before dispatch. Pass the signal to the active http.request and destroy the request on abort. Provider-preflight reads, model-load/contention retry delays, recursive retries, chunk transitions, and batch yields must all observe the signal. No later chunk/request may start after abort. Preserve OPENAI_COMPATIBLE_REQUEST_TIMEOUT for provider-owned request deadlines. Local queue/socket disposal is provable; remote work already accepted before socket destruction may continue provider-side.
Ollama Pass signal through #embedOllama() into the existing Ollama.embed(..., {signal}) native request path. Preserve the existing PROVIDER_TIMEOUT distinction. Local request destruction is provable; completion state inside the Ollama server after disconnect is not asserted.
Gemini Pass {signal} as the installed SDK's per-call request options for both embedContent() and batchEmbedContents(). If the SDK wraps an abort while the caller signal is aborted, restore the Error-valued caller reason or the bounded AbortError fallback defined above. The installed SDK explicitly defines AbortSignal as client-only cancellation: local fetch rejects, but remote computation and billable usage may continue. It also owns an internal controller/listener that Neo cannot explicitly remove. Both remain classified residuals, not “true provider cancellation” and not “unsupported.”

Consumer and Cadence Contracts

1. Cached Memory Core health canary

  • Keep buildEmbeddingWriteCanaryBlock() as a regular healthcheck diagnostic.
  • Replace its non-cancelling Promise.race() use with an AbortController whose deadline is the existing memoryCoreConfig.healthcheck.embeddingWriteCanaryTimeoutMs / injected timeoutMs.
  • Preserve the existing healthy-only cache and its cadence. This ticket does not turn healthcheck traffic into action admission.
  • Timeout/abort returns a bounded failed/inconclusive canary block; it does not throw a heal action into existence.

2. Orchestrator freeze re-probe

  • Keep the existing due-only cadence: runFreezeReprobeCycle() calls the live probe only after the durable freeze record passes its back-off/cap pre-decision. No probe runs per poll when deferred or contained.
  • Add AiConfig.orchestrator.recoveryActuator.freezeReprobeTimeoutMs as the orchestrator-owned deadline leaf, default 30 seconds, env-bound as NEO_RECOVERY_ACTUATOR_FREEZE_REPROBE_TIMEOUT_MS.
  • Read that leaf inside probeFrozenCollectionHealth(), create one controller per due probe, and pass its signal plus a bounded operation label to embedTexts().
  • Own-deadline expiry, upstream abort, either provider-timeout code, and other provider failure all become inconclusive evidence. The existing decider therefore stays frozen; the raw probe cannot unfreeze or select another action.

No new action-admission probe is added. If a future classifier needs one, it requires its own evidence/cadence contract.

Contract Ledger

Dimension Contract
Invocation Existing healthcheck canary calls and due-only freeze re-probes; never ordinary restore, per row, per importer batch, or every orchestrator poll.
Ownership Callers own controllers/deadlines; TextEmbeddingService propagates and restores the caller-owned abort shape where an SDK wraps it; provider branches own local queue/transport disposal; ADR-0025/0027 deciders retain action authority.
Inputs Required explicit provider plus optional {signal, operationLabel}; no provider is inferred or switched by cancellation.
Outputs Original embedding result on success; structural abort/timeout error on cancellation; consumer-specific bounded diagnostic receipt after mapping.
Deadline SSOT Health canary uses its existing Memory Core healthcheck leaf; freeze re-probe uses the new orchestrator recovery-actuator leaf, each read at its use site.
Queue semantics A queued OpenAI-compatible task is removable before dispatch; an aborted task can never open a later request or chunk.
In-flight semantics Locally-owned active requests/fetches are aborted/destroyed; Gemini remote continuation is admitted unknown.
Retry semantics Retry delays, recursive retries, provider preflight, and batch yields are abort-aware; cancellation consumes no further retry budget.
Concurrency Abort cleanup settles before the local queue admits the next recovery batch witness; no Neo-owned stale listener may affect a later task.
Error taxonomy Upstream abort is distinct from provider timeout and consumer probe timeout. Provider timeout codes remain branch-specific: OpenAI-compatible OPENAI_COMPATIBLE_REQUEST_TIMEOUT; Ollama PROVIDER_TIMEOUT; consumer-owned deadline EMBEDDING_PROBE_TIMEOUT. Gemini restores an Error-valued caller reason after SDK wrapping, with bounded AbortError / ABORT_ERR fallback.
Cleanup Neo-owned timers, listeners, and queue entries settle/remove exactly once on success, abort, timeout, and provider failure. Gemini SDK-owned listener lifetime is an explicit residual and is not represented as Neo-cleanable state.
Idempotency Re-aborting or observing an already-aborted signal has one settlement and no duplicate queue removal/destroy side effect.
Observability Provider, operation label, local phase (preflight/queued/in-flight/retry/yield), classification, timing, and bounded redacted reason. No input text or secret material.
Versioning Third argument is additive; all two-argument callers remain valid and retain behavior. Existing provider-timeout codes remain compatible.

Acceptance Criteria

  • Both embedding methods accept (input, explicitProvider, {signal, operationLabel}) without breaking two-argument callers.
  • Upstream abort, provider timeout, and consumer probe timeout retain the distinct structural taxonomy above; OpenAI-compatible and Ollama timeout-code compatibility is pinned separately.
  • OpenAI-compatible: already-aborted and queued-aborted calls open zero provider requests; in-flight abort destroys the request; preflight/retry/yield/chunk boundaries start no later work.
  • OpenAI-compatible shared-provider witness proves local queue disposal settles before the next recovery batch is dispatched.
  • Ollama specs prove the upstream signal reaches and destroys the native embedding request without becoming PROVIDER_TIMEOUT.
  • Gemini specs prove the signal reaches SDK request options; an SDK-wrapped abort re-surfaces the exact Error-valued caller reason or the bounded AbortError fallback; a non-caller-abort SDK error stays unchanged. Docs and receipts preserve the remote-compute/billing and SDK-owned-listener residuals.
  • The cached health canary aborts underlying local work at its existing deadline and retains its existing diagnostic/cache role.
  • A due freeze re-probe reads the new deadline leaf at the use site, aborts transport at expiry, and maps EMBEDDING_PROBE_TIMEOUT, upstream abort, either provider-timeout code, and other provider failures to fail-closed stay-frozen evidence.
  • Deferred/contained freeze records, ordinary preserved-vector restore, and importer rows/batches perform zero probes.
  • Neo-owned timers/listeners/queue entries are cleaned on success, abort, timeout, and provider failure; tests do not claim control over the installed Gemini SDK's internal listener.

Out of Scope

  • A new action-admission canary
  • Scheduling or executing re-embedding
  • Provider warming/restart policy
  • Guaranteeing cancellation of Gemini server-side compute or billing
  • Normalizing existing provider-timeout codes across providers
  • Refactoring unrelated withTimeout() consumers
  • Reintroducing any restore-wide provider gate

Avoided Traps

  • Timeout without cancellation — releases the caller, not scarce provider capacity.
  • One generic provider claim — hides materially different queue/socket/fetch residuals.
  • Probe per poll, batch, or row — turns evidence collection into the dominant workload.
  • Abort as action authority — violates diagnostics/actuation separation.
  • Reusing one deadline across owners — couples healthcheck cadence to recovery cadence and violates config ownership.
  • Claiming remote cancellation from local disconnect — especially false for the Gemini SDK's documented client-only signal.
  • Claiming cleanup authority over SDK internals — Neo can prove only the lifecycle of state it owns.
  • Timeout-taxonomy scope creep — provider-code normalization is a separate compatibility decision, not hidden inside transport abortability.

Related

  • #13458 — bounded outer health waits
  • #14036 / PR #14047 — provider request-duration bounds
  • PR #12722 / PR #12818 — upstream abort and provider-timeout precedent
  • #15689 / PR #15690 — preserved-vector restore stays provider-free
  • #15693 — orchestrator-governed recovery action consumer
  • ADR-0019, ADR-0025, ADR-0027

Live latest-open and successor sweep found no equivalent open ticket or PR. This ticket closes the remaining signal-propagation gap across the two live embedding-probe consumers.

Origin Session ID: cb60301d-74a4-4024-b80d-2f7efdbf9cd1

Retrieval Hint: "abortable embedding health canary freeze reprobe OpenAI queue Ollama Gemini client-only cancellation"