One morning this month I opened the status page for my runtime watchdog for a live agent — a small scheduled service that watches how an agent behaves in production and scores what it sees. The heartbeat was green. It had been green all night. It had been green the entire previous day.
And the watchdog had done no useful work in all that time.
Let me define the term before we go further, because everything in this post hangs on it. A heartbeat is a timestamp that a scheduled job writes every time it runs — a little note that says “I was alive at 09:15”. My watchdog beats every fifteen minutes. On every beat, an orchestrator runs, works out which checks are due, hands each one off as a background job, stamps its heartbeat, and exits cleanly.
That is exactly what it did, all day. The orchestrator ran. It enqueued its jobs. It stamped its heartbeat. And every single job it enqueued died the moment it started. By its own definition, the heartbeat was correct — the orchestrator really did run every fifteen minutes. It was also completely useless, because the thing I actually cared about had been dead for a day.
A status light only ever tells you that the reporter ran. It says nothing about whether the work it reported on survived.
That gap — between what a green light says and what we hear — is the whole lesson.
Why try/catch could not save me
In Apex, the usual way to handle errors is a try/catch block: you wrap risky code in try, and if it throws an error, the catch block runs and you can log what happened. Most monitoring is built on this idea — catch the failure, write it somewhere visible.
My jobs died with a platform-level “Salesforce System Error”. This is a class of failure that the platform raises from underneath your code, and no try/catch intercepts it. The transaction is torn down entirely — and your catch block, your logging call, your error record are all part of the wreckage. Nothing you wrote gets a chance to speak.
The jobs themselves were Queueables — Apex’s mechanism for handing a unit of work to a background process, each running in its own transaction. Each one started, hit the system error, and vanished without a trace my code could leave. Any monitoring strategy that depended on the failing code reporting its own failure was blind by design.
Diagnose by contrast: what did not break?
Here is the debugging habit that actually found the cause. A sibling service in the same org runs on the same schedule and performs the same DML — DML being the database operations, the inserts and updates. It survived the whole outage without a single failure.
Same platform, same DML, same timing — one dead, one healthy. So the DML was innocent. The one real difference between them: the survivor bucketed its state per day and per hour — fresh values in fresh compartments — while the casualty accumulated a single running value that it updated on every observation, forever.
That contrast pointed straight at the arithmetic on the running value, not at anything the jobs did to the database.
When you are bisecting a failure, do not only ask what broke. Ask what nearby thing did not break — and why. The survivor is evidence too.
The Decimal-scale time bomb
The running value was an EWMA — an exponentially weighted moving average. It is a one-line way to keep a smoothed score where recent observations count more than old ones:
ewma = (alpha * observed) + ((1 - alpha) * ewma);Harmless-looking. But in Apex, when you multiply two Decimal values, the result’s scale — the number of digits after the decimal point — is the sum of the two input scales. Multiply a scale-6 number by a scale-4 number and you get scale 10. An EWMA that never rounds therefore gains digits on every single observation. The number is still mathematically tiny; its representation just keeps growing.
For a while, nothing visible happens. In my case, after roughly 380 observations in one series, the arithmetic had grown until the transaction died with that uncatchable system error. Three hundred and eighty observations is not an exotic number — it is simply a series that has been running for a while. Every org that keeps a never-rounded running Decimal will reach that history length eventually. The fuse is not load or bad luck. The fuse is time.
Two fixes I reverted — and the one I kept
Before I had measured the real cause, I tried two plausible fixes. Both looked reasonable. Both got reverted the moment the measurement came back and showed the actual mechanism, because neither of their stated reasons was true.
I want to be firm about this, because it is tempting to leave a workaround in place once things go quiet: a workaround whose stated reason is false is a lie in the code. The next person — often you, six months later — reads the comment, believes it, and builds on a wrong model of the system.
The real fix is one honest line — bound the scale explicitly after every EWMA update:
ewma = ((alpha * observed) + ((1 - alpha) * ewma))
.setScale(6, RoundingMode.HALF_UP);The scores live between 0 and 1, and six decimal places is already more precision than any threshold in the system uses — so nothing meaningful is lost, and the scale can never grow again. Alongside it sits a guard test that replays a long series of observations, so this particular bomb cannot quietly return.
Receipts: let absence be the detector
The fix repairs one bug. The pattern below repairs the monitoring, and it works even for failures your code never sees coming.
- The scheduler records what it enqueued. On every beat, the orchestrator writes down exactly which units of work it handed off — a receipt of intent.
- Every unit of work writes a completion marker as the last act of its own transaction. Not first, not midway — last, so the marker only exists if the whole transaction survived.
- A unit that dies writes nothing. Even an uncatchable death leaves the same evidence: silence. You compare receipts against markers, and the absence of a marker is your detector. Nothing had to survive the crash to tell you about it.
Receipts tell you that something died. For why, ask the platform itself. Salesforce keeps its own record of every background job in AsyncApexJob, and it writes there even when your code could not:
SELECT Id, Status, ExtendedStatus
FROM AsyncApexJob
WHERE JobType = 'Queueable' AND Status = 'Failed'The ExtendedStatus field carries the error text. Receipts detect, AsyncApexJob explains — the platform records what your own catch block cannot.
Show two timestamps, not one
One last change, and it is the cheapest of all. On any status page, show “latest beat” and “latest fully completed beat” as two separate facts. During my outage those two timestamps differed by a full day — and that difference was the only symptom visible from the outside. A single merged “last run” timestamp would have hidden it, and did.
Your next step
You do not need a watchdog or an agent to use this today. Pick one green light in your own org — a scheduled job, an integration status, a sync indicator — and ask it a single question: what would still say OK if the thing you stand for had died? If the honest answer is “I would stay green”, you have found a heartbeat that only measures its own reporter. Add one completion marker, written as the last act of the real work, and let absence do the detecting. It is a small change — and it is the difference between a dashboard that is green and a system that is actually alive.