A job that runs on time and does nothing

A monitor that lies to you is worse than no monitor. Not slightly worse. Worse in kind, because it answers the question you would otherwise have gone and checked yourself, and it answers it in the direction that requires no action.

I spent a few days building a dead man's switch for scheduled jobs, and the interesting part was not the switch. It was that the two worst bugs in it were the same bug in opposite directions. Both times the monitor claimed to know something it had no evidence for.

The problem with three states

Every cron monitor on the market models three states: it ran, it failed, it is late. Healthchecks.io, Dead Man's Snitch and Cronitor have all done this well for years, so this is not a gap in the market so much as a gap in the model.

Three states are enough for a job that either works or falls over. They are not enough for a job that processes a queue, because such a job has a fourth outcome that looks exactly like success: it ran, on time, and did nothing.

The outlined states have no equivalent above them. This is a gap in the model rather than in the tooling.

Sometimes that is correct. An inbox is legitimately empty overnight. But a worker that has silently detached from its queue reports exactly the same thing, forever, in green. Exit codes cannot tell these apart, because both are zero. Logs cannot, because both are quiet. The only thing that can is the job saying how many items it handled, and treating a reported zero as information rather than as an absence of information.

That gives you a state worth having. Call it quiet: succeeded repeatedly, did nothing at all, for long enough that an empty queue is no longer a plausible explanation.

The first bug: unknown accumulating toward an alarm

Once a job reports a count, quiet is a streak counter. Count consecutive runs that did nothing, and alarm past a threshold. That is about four lines of code and it is wrong.

The job I tested it against reports a count on one path and nothing on another. When its pre-check found no work it exited early with a count of zero. When there was work to do it ran the real thing, which had no idea the monitor existed and reported nothing at all.

So the sequence of runs alternated: a zero, then silence, then a zero, then silence.

Both counters see identical input. The upper one is counting evidence it does not have.

A counter that increments on zero and ignores anything else sees only the zeroes. The streak climbs forever and never resets, so a job that was working perfectly gets convicted of doing nothing. The runs where it did the most work are the ones that contributed nothing to its defence.

The fix is one line and the reasoning behind it is the whole point:

export function nextQuietRuns(current: number, run: Run): number {
  if (!run.ok) return current;
  if (run.did_work === null) return 0;   // unknown BREAKS the streak
  return run.did_work === 0 ? current + 1 : 0;
}

The counter is not a streak of runs that did nothing. It is a streak of runs known to have done nothing, and an unreported run breaks it, because the absence of a count is not evidence of an absence of work.

Unknown must never accumulate toward an alarm. If you find yourself defaulting a missing measurement to zero anywhere in a monitoring path, you have just invented the number your alarm fires on.

The second bug: a date somebody else gave you

The opposite failure arrived from the opposite direction, and it was mine too.

A monthly job registered today has not missed anything, so a newly declared job sits waiting rather than alarming. That is correct, and it is also a hole: a job that was already broken looks healthy for its entire first cycle. For a monthly job that is a month of false calm, which is a month during which the tool is worse than useless because it is actively reassuring.

The fix was to let a declaration carry the job's real last run, taken from its schedule or its logs, so the clock starts from reality. Declare a monthly job that last worked seven weeks ago and it goes red immediately instead of next month. That worked, and it is the feature that made a set of dead jobs findable on day one rather than on day thirty.

Then I looked at the board and found a job reading working, ran 7 days ago that the collector had never received a single check-in from. Not one. The date on screen was the anchor I had supplied. The run history behind it was empty.

Every part of that was doing what I wrote. The anchor filled the last-run timestamp, nothing had marked the job as failing, so it fell through to the healthiest state available. The strongest claim on the board was resting entirely on a date I had typed in myself.

An anchor is somebody else's word about the past. It is good enough to start a clock and not good enough to certify a job as healthy, so those two uses had to come apart:

// A seeded anchor says when this agent last ran according to somebody else. It
// is worth trusting to start the clock. It is not evidence the collector has
// ever seen this agent do anything, so it must not buy a confident green.
if (!observed(a)) return "pending";

Now it reads last ran 7 days ago per its schedule, waiting for its first check-in, which is longer, uglier, and true. Overdue still wins, so the anchor keeps doing the job it was added for. It just cannot manufacture a green any more.

The same mistake twice

Written out, the two are one mistake with two faces. A missing count became a zero, and a supplied date became an observation. Both times a gap in the evidence got filled with the most convenient available value, and both times the result was a confident display backed by nothing.

The direction is almost incidental. One would have woken me up about a job that was fine, the other would have let a dead job sit green indefinitely. The false alarm is the one you notice, so it gets fixed. The false calm is the one that matters, and by construction nothing tells you about it.

That is a hard thing to test for, because the failure mode of a monitor is silence, and silence is what it looks like when everything is fine.

Two practical consequences

The first is to make absence visible instead of representing it as the absence of a warning. Every job on the board is drawn as a trace, every row ends at the same NOW line, and the gap between the last check-in and now is hatched rather than left empty. A dead job is a row that stops short of the edge, and you find it by looking.

Idle and quiet are the same shape. What separates them is how long it has gone on.

A status lamp cannot do this. It gives a dead job and a quiet one the same colour, because it has one bit to say it with.

The second is to break it on purpose once it is installed. Stop the job and confirm it goes missing. A check that has never failed has not been tested, it has only been installed, and every wrong way to wire up a heartbeat produces a board that stays reassuringly green. Check in at the start of the job rather than the end and a crash halfway through looks perfect forever. Check in unconditionally on the line after the real command and it says fine every time. Send a hardcoded count of one and you have defeated the entire idea, because a number that can never be zero means a job that has stopped doing anything still reads as busy.

If you cannot count it, leave it out. Not reported is honest. Invented is not.