← Machinery Machinery · 01

Five nets, not one
clever check.

A rented GPU can stop being useful in five structurally different ways, and the meter runs through all of them. Each way is caught by a separate watchdog with a narrow gate, because the checks that would catch the other four are exactly the checks that stay warm during this one.

The argument

Each net is blind to what the other four catch.

A watchdog is a question asked on a timer, and the question fixes what it can possibly see. Ask “has this machine gone quiet?” and you never notice the one that was never noisy: with no first report to go stale from, its silence looks identical on the first tick and the thousandth. Ask “has it ever reported?” and you never notice the one that reported for an hour and then stopped. Neither question reaches an instance that exists at the provider but was never recorded here — both start from an inventory it is not in.

The tempting move is to merge them into one health check with one threshold. It fails in the expensive direction: a merged check has to be permissive enough not to destroy a machine that is legitimately in a slow phase, and at that threshold it stops catching the fast, cheap-to-detect failures the narrow checks were built for. Five questions, five gates, five cadences.

LIFECYCLE OF ONE MACHINE → PROVIDER SIDE REQUESTED BOOTING SERVING SILENT NO RECORD HERE PROVISIONING DEADLINE FIXED WHEN THE REQUEST IS PLACED NEVER REPORTED NO FIRST SIGNAL WENT SILENT REPORTED ONCE, THEN STOPPED LOST MARKER NEVER DESTROYS ORPHAN SWEEP THEIR LIST, NOT OURS
Five nets over one lifetime, and they barely overlap. The highlighted one catches the expensive case — an instance that is genuinely up, genuinely billing, and doing nothing. The fifth sits on the other side of the divider because a machine we have no record of can only be found by reading the provider's list, never our own.
How it diedWhat still looks fineThe net that sees it
Provisioning stalls before the machine ever registers itself Nothing has gone stale — there is no report to age. The provisioning deadline, fixed at the moment the request was placed.
The instance boots, but its start-up never completes The provider reports a running instance, correctly. The never-reported net, once the boot window has elapsed.
The worker process dies while the instance keeps running Every check that talks to the provider succeeds. The instance really is up. The went-silent net — and only that one.
The network path to the machine drops The machine itself may be perfectly healthy behind the break. The lost marker: it flags the machine and stops routing to it, and deliberately does not destroy.
An instance exists at the provider with nothing recorded here Every net that starts from our own inventory, because it is not in it. The orphan sweep, comparing the provider's list against ours.
Cadence is part of the design The four nets that read our own records ask every minute; the sweep that reconciles against the provider's list asks every five, because it means listing every instance at every provider and the failure it catches is not getting more expensive by the second. A watchdog's period is chosen against the cost of the thing it prevents, not set globally.
The money chain

One path here has no gate at all, on purpose.

Idle machines are the largest avoidable line on a GPU bill, and the chain that handles them is the only one that calls a provider to turn something off. Every thirty seconds a short live utilisation sample decides one of two things about each running machine. If it is quiet, two timers are armed together: a stop timer and, behind it, a destroy timer. If any traffic was seen, both timers are cleared.

The clearing step is the one place in the chain with no condition on it. Everywhere else a gate that misfires fails safe: the worst outcome is a machine that keeps running and keeps costing money until someone notices. On the clearing path a gate can only fail dangerous. Make clearing conditional on the machine being in a particular recorded state, and a machine that has drifted out of that state — for any reason, including one nobody anticipated — can never clear its timers again. It keeps serving live traffic with a destroy timer counting down against it, and no amount of observed work helps, because the observation is never permitted to take effect. So the rule is blunt: traffic clears both timers, whatever the recorded state says.

UTILISATION SAMPLE EVERY 30 SECONDS QUIET ARM BOTH TIMERS STOP, THEN DESTROY TRAFFIC SEEN DISARM: NO GATE ANY TRAFFIC CLEARS IT WHATEVER THE STATE SAYS CLEARS BOTH STOP WATCHDOG EVERY 30 SECONDS DESTROY WATCHDOG EVERY 30 SECONDS STOPPED DISK KEPT DESTROYED NO BILL
Every box on the acting path checks something before it moves, and a gate there fails safe. The highlighted box is the exception: clearing an armed timer is unconditional, because a condition on that path can only fail in the direction that destroys a working machine.

Both timers are armed together and fall due at different times, so the reversible action always precedes the irreversible one — a stopped instance keeps its disk, a destroyed one is gone. The intent is recorded before either provider call goes out, and the component that continuously reconciles our inventory against the provider only acts on machines whose recorded intent is to be running, so a deliberate stop is never quietly undone by the next reconcile.

The check that is easy to leave out A destroyed machine still has a record, and a scan that forgets to exclude those keeps feeding dead entries into the rest of the chain: timers re-armed against machines that no longer exist, and eventually a stop or destroy call for an instance the provider deleted long ago. The calls fail harmlessly and nobody is paged, which is exactly why this omission is so easy to keep.
Before anything is destroyed

Most of a reaper is the list of reasons not to fire.

The destroy call itself is one line. Everything above it is refusals, and each refusal exists because the naive version of the check destroys something it should not. This is the order every machine-level reaper walks before it is allowed to act.

  1. Refuse — it is not ours to endOnly machines this platform provisioned are candidates. A box that registers itself and is administered by someone else is observed, listed and routed around, but never destroyed, at any cadence. The gate is the presence of our own provisioning record rather than a list of known providers: a list you forget to extend when a provider is added silently stops reaping, which is the expensive way to be wrong.
  2. Refuse — the record says it is already goneA destroyed machine is skipped rather than re-evaluated, so nothing re-arms a timer against an instance that no longer exists.
  3. Refuse — work is pinned to itA machine holding a job is skipped regardless of how quiet it looks. Silence during a long weight download or a long training step is normal; the job releases the machine when it is done.
  4. Refuse — it was never expected to reportSome workload shapes never send a heartbeat by design. They are exempt by class, not by a longer timeout — a longer timeout only delays the wrong answer.
  5. Refuse — it is still inside its grace windowBoot windows run from the moment this platform first observed the machine, not from the timestamp the provider reports. The two differ, and the difference is comfortably long enough to kill a machine that is still pulling its image — which provisions a replacement that is also still pulling its image.
  6. Act — every gate clearedThe intended end state is recorded before the provider call is made, so nothing is ever left believing a machine is gone while it is still billing. A call that fails leaves the machine in a transitional state rather than back in the pool — visible, and picked up by a separate watchdog rather than silently retried.
Why the gate stays narrow

A reaper and an operator ask different questions about the same machine.

There are three independent ways to believe a machine is alive: the provider answers that the instance exists, our own control plane records a successful poll of it, and the worker on the machine reports for itself. Folding all three into one freshness value is the right answer to an operator's question — is anything at all still in contact with this machine? — and the machine-detail API returns exactly that, because on machines that register themselves one of the three channels is never written by anybody and would otherwise read permanently stale.

It is the wrong answer to the reaper's question, which is narrower: is the process that was supposed to be doing work still doing it? Only the worker's own report answers that. The other two channels are written by something other than the workload, so they stay warm in the exact failure the went-silent net exists for — the instance is up, the provider is happy to keep charging for it, and the thing being paid for has been dead for a quarter of an hour. Widen that gate to the composite and nothing ever destroys a wedged, billing machine again.

WORKLOAD DIES INSTANCE KEEPS BILLING PROVIDER POLL PROVES THE INSTANCE EXISTS CONTROL-PLANE PROBE WRITTEN BY US, NOT THE BOX WORKER SELF-REPORT THE ONLY CHANNEL THAT COOLS STILL WARM STILL WARM GOES COLD DESTROYED BILLING ENDS
Two of these three channels are kept warm by something other than the workload — one by the provider, one by us. A watchdog that required all three to go cold would never fire in the one case it was built for.

Thresholds follow the same logic. A machine that reported and then stopped gets a quarter of an hour, with a hard floor under that value so a misconfiguration cannot shrink it to something that kills healthy machines mid-request. A machine that never reported at all gets half an hour from first observation, because that window has to cover a cold image pull on a slow link. A machine that is merely unreachable is marked and taken out of rotation, never destroyed: the fault may be in the path rather than in the machine.

Reference

Every watchdog, its cadence, and its blast radius.

Read the third column as an audit: it is what the decision is actually made on. Five of these can destroy a machine; the last four cannot touch one at all, which is why they are allowed much cruder rules.

WatchdogAsks everyWhat its gate readsWhat it can do
Idle scanner30 sA short live utilisation sampleArms or clears the stop and destroy timers
Auto-stop30 sThe stop timer, plus whether work is pinned to the machineStops the instance; the disk survives
Auto-destroy30 sThe destroy timer, same work gate, machine already stopped or idleDestroys the instance
Boot failure60 sWhether the machine has ever reported, against its boot window; start-up failure is re-confirmed live across ticks before actingDestroys
Provision deadline60 sThe deadline fixed when the request was placedDestroys
Went silent60 sThe worker's own report clock, and deliberately nothing elseDestroys
Lost detector60 sAll three liveness channels togetherMarks the machine lost and stops routing to it
Orphan sweep5 minThe provider's instance list against our inventoryDestroys instances older than the grace window with nothing recorded here
Pool idle release15 minA pool's own idle clockReleases the pool; never touches a machine
Queue-wait timeout60 sA queued request's age against its deadlineFails the request
No-consumer detector60 sWhether anything is listening for a pool's workFails the request rather than letting it wait forever
Stuck-dispatch reset60 sA request's provisioning counterResets it so the request can be dispatched again
The pairing that makes the table safe Every row that reads a machine's state also reads whether this platform is the component responsible for maintaining that state. A gate that reads a state field without that pairing is asking about a machine nobody answers for, and it will act on whatever value happened to be written last. That is the subject of the next page in this series.
When a reaper cannot classify a failure These detections do not each carry their own destroy logic. They land in one append-only ledger and route to a small fixed set of outcomes: do nothing, retry the same machine, fail the job, replace the machine, stop using that hardware configuration, halt provisioning, or alert a human. A failure class the routing table does not recognise maps to the last of those, never to a retry — retrying an unclassified permanent failure spends real money to reach the same answer.

Run cheap GPUs without the bill shock.

Request early access →