Split out of #16880 after PR #16943 shipped a fix for it that broke that ticket's own subject. Reverted at 9bcb7623f2; recording the defect properly rather than losing it with the commit.
The defect
provider_activity_log rows are durable. The limiter that admits them is per-process and in-memory. So a process that dies mid-flight leaves rows with completed_at IS NULL that nothing will ever complete — the thing that would have completed them is gone — and the live-demand projection counts them as current work.
Measured shape: {cap: 1, executing: 1} immediately after a restart that admitted nothing, and it never clears.
Same class as the rest of the #16706 incident: a durable record outliving the thing it describes, read as current.
Why the obvious fix is wrong — evidence, not prediction
I shipped it and @neo-gpt falsified it.
Stamping a module-scope PROCESS_EPOCH on each row and filtering the projection to process_epoch = @process_epoch means a reader sees only rows it wrote. An MC or Orchestrator reader then reads Knowledge Base live demand as executing: 0 — and cross-service admission visibility is the entire subject of #16880. It closes a latent boundary by opening a live one.
A same-module test cannot catch this. Two services in one process share one epoch, so the writer-A / observer-B split never occurs. My arm passed for a reason unrelated to what it claimed.
What a correct fix needs
A liveness signal, not a stronger filter. The requirement is asymmetric:
- exclude rows whose owning process is gone
- include rows whose owning process is live, including other services
Time alone cannot separate those. A dead process's uncompleted row and a live process's slow row are identical from the ledger — which is precisely the waiting-versus-broken conflation that made this incident unreadable for weeks. Any threshold on age reproduces it.
So the epoch has to be joined against something that says whether that generation is still running: a heartbeat table, a registration row with a TTL, or an equivalent. That is a shared-ledger schema decision and it deserves its own design rather than a patch.
Acceptance criteria
Deltas
- Latent, not live. It needs a process death mid-flight, and the misreport then persists for that store. It is not what pegged the cores on the affected plane.
- #16880 keeps the per-process KEYING fix (
service::provider), which is unaffected and shipped in PR #16943.
Split out of #16880 after PR #16943 shipped a fix for it that broke that ticket's own subject. Reverted at
9bcb7623f2; recording the defect properly rather than losing it with the commit.The defect
provider_activity_logrows are durable. The limiter that admits them is per-process and in-memory. So a process that dies mid-flight leaves rows withcompleted_at IS NULLthat nothing will ever complete — the thing that would have completed them is gone — and the live-demand projection counts them as current work.Measured shape:
{cap: 1, executing: 1}immediately after a restart that admitted nothing, and it never clears.Same class as the rest of the #16706 incident: a durable record outliving the thing it describes, read as current.
Why the obvious fix is wrong — evidence, not prediction
I shipped it and @neo-gpt falsified it.
Stamping a module-scope
PROCESS_EPOCHon each row and filtering the projection toprocess_epoch = @process_epochmeans a reader sees only rows it wrote. An MC or Orchestrator reader then reads Knowledge Base live demand asexecuting: 0— and cross-service admission visibility is the entire subject of #16880. It closes a latent boundary by opening a live one.A same-module test cannot catch this. Two services in one process share one epoch, so the writer-A / observer-B split never occurs. My arm passed for a reason unrelated to what it claimed.
What a correct fix needs
A liveness signal, not a stronger filter. The requirement is asymmetric:
Time alone cannot separate those. A dead process's uncompleted row and a live process's slow row are identical from the ledger — which is precisely the waiting-versus-broken conflation that made this incident unreadable for weeks. Any threshold on age reproduces it.
So the epoch has to be joined against something that says whether that generation is still running: a heartbeat table, a registration row with a TTL, or an equivalent. That is a shared-ledger schema decision and it deserves its own design rather than a patch.
Acceptance criteria
executing: 1and observer B, in a different module, reports the same row asexecuting: 1. A same-module two-service arm does not satisfy this and must not be offered as if it did.0reads as "admission is closed", which is the most alarming possible value.Deltas
service::provider), which is unaffected and shipped in PR #16943.