Context
The #14085 truncate-to-context floor (merged via #14092) recovers oversized Memory Core documents by embedding a context-bounded prefix. truncateToEmbedTokenBudget derives the byte budget from the shared bytesToTokens heuristic (~3 bytes/token).
Surfaced in @neo-opus-grace's #14092 review (non-blocking follow-up #2).
The Problem
bytesToTokens (bytes ÷ 3) under-covers dense content — CJK / emoji / dense handoffs run ~2.7 (or fewer) chars/token and more tokens per byte than the /3 heuristic assumes. So for a dense oversized document, the truncated prefix (sized at maxTokens × ~3 bytes) can still exceed the real provider token budget → the provider rejects it → it degrades to unrecoverable. The floor silently under-serves emoji/CJK-heavy memories. (Clean degradation — not a bug — but the floor under-covers them, partly defeating its purpose for dense docs.)
The existing shave-loop doesn't help: it re-checks against the same under-estimating bytesToTokens, so it trusts the heuristic that's wrong for dense content.
The Fix
Apply a safety-margin shave (~0.9×) to the truncate budget so a dense doc's prefix lands safely under the real budget:
maxBytes = floor(maxTokens × bytesPerToken × EMBED_TRUNCATE_SAFETY_FACTOR) with a named EMBED_TRUNCATE_SAFETY_FACTOR ≈ 0.9. (Alternatively a real-token re-check, but the margin is the minimal-blast fix.) Existing tests still pass (the margin only makes truncation slightly more aggressive).
Acceptance Criteria
Out of Scope
- Truncated-row count/flag visibility (Grace's #14092 follow-up #1 — needs the
embedFn-wrapper → repair-counts plumbing; separate follow-up; also feeds #14084's accepted-loss taxonomy as a recovered-with-truncation category).
- A full provider-tokenizer integration (the margin is the heuristic mitigation).
Related
- #14085 / #14092 (the merged floor + the review that surfaced this), #14084 (accepted-loss taxonomy that #1 feeds), #14026 (the data-integrity epic).
Authored by Ada (Claude Opus 4.8, Claude Code).
Context
The #14085 truncate-to-context floor (merged via #14092) recovers oversized Memory Core documents by embedding a context-bounded prefix.
truncateToEmbedTokenBudgetderives the byte budget from the sharedbytesToTokensheuristic (~3 bytes/token).Surfaced in @neo-opus-grace's #14092 review (non-blocking follow-up #2).
The Problem
bytesToTokens(bytes ÷ 3) under-covers dense content — CJK / emoji / dense handoffs run ~2.7 (or fewer) chars/token and more tokens per byte than the /3 heuristic assumes. So for a dense oversized document, the truncated prefix (sized atmaxTokens × ~3bytes) can still exceed the real provider token budget → the provider rejects it → it degrades tounrecoverable. The floor silently under-serves emoji/CJK-heavy memories. (Clean degradation — not a bug — but the floor under-covers them, partly defeating its purpose for dense docs.)The existing shave-loop doesn't help: it re-checks against the same under-estimating
bytesToTokens, so it trusts the heuristic that's wrong for dense content.The Fix
Apply a safety-margin shave (~0.9×) to the truncate budget so a dense doc's prefix lands safely under the real budget:
maxBytes = floor(maxTokens × bytesPerToken × EMBED_TRUNCATE_SAFETY_FACTOR)with a namedEMBED_TRUNCATE_SAFETY_FACTOR ≈ 0.9. (Alternatively a real-token re-check, but the margin is the minimal-blast fix.) Existing tests still pass (the margin only makes truncation slightly more aggressive).Acceptance Criteria
truncateToEmbedTokenBudgetapplies a named safety-margin factor so the truncated prefix's estimated tokens are ≤margin × maxTokens.Out of Scope
embedFn-wrapper → repair-counts plumbing; separate follow-up; also feeds #14084's accepted-loss taxonomy as a recovered-with-truncation category).Related
Authored by Ada (Claude Opus 4.8, Claude Code).