← Machinery Machinery · 04

Nine branches.
Seven of them say
stay put.

One GPU, several kinds of work and nobody scheduling it: the machine binds itself to whichever pool has work — and almost every branch of the loop that could move it is a reason not to.

The loop

The rotation, when nothing stops it. live

A GPU serving nothing costs exactly what a GPU serving everything costs. So a machine here does not wait to be told what to do. It binds to a pool that has work, stays bound while work keeps arriving, and when the work stops it goes looking — first for another pool, then for a queued training run. Every one of those steps is a decision it is allowed to decline, and most of this page is about when it declines.

Three states, two directions of travel. A machine is unbound, serving one pool, or training. It moves between them on its own, on a tick, with no operator in the path and no central scheduler handing out assignments.

WORK KEEPS ARRIVING · IDLE CLOCK RESETS OR A HIGHER-PRIORITY BACKLOG REBINDS IT · DWELL FLOOR 60 S UNBOUND NOTHING TO SERVE SERVING A POOL BOUND TO ONE POOL TRAINING ENGINE DRAINED A POOL HAS WORK PRIORITY ORDER NO OTHER WORK ENGINE QUIET 20 MIN LAST RESORT IDLE 5 MIN · ANOTHER POOL HAS WORK RUN ENDS · ENGINE RESTORED FROM ITS PRE-DRAIN RECORD
Training is the last resort, not the default. The machine only drains a serving engine after it has already failed to find any pool with pending work — serving outranks a queued run, because someone is waiting on one and nobody is waiting on the other.

Four windows govern the whole loop. They are deliberately unequal: the one that guards the destructive move is four times longer than the one that guards a harmless one.

ClockWindowWhat it decides
Idle before switching pools 5 min How long a bound machine tolerates silence before it goes looking for other work.
Minimum dwell 60 s The floor between two moves, so a change in priority cannot make the machine thrash.
Engine quiet before training 20 min How long the inference engine must be genuinely silent before the machine is willing to tear it down.
Training pool release 15 min How long a training pool sits idle before the machine is handed back to serving.
Why the long window is anchored on the engine An empty job queue does not mean an idle GPU. Traffic can reach a self-hosted engine directly, over the private network, without ever passing through the platform's counters. So the twenty-minute clock watches the engine's own activity rather than the queue depth — a busy machine cannot be drained by a queue that merely looks quiet.
The decision order

Nine branches. Two of them move anything.

On each tick the machine walks this list top to bottom and takes the first branch that matches. It is easier to read as a list of reasons not to move, because that is what it mostly is.

  1. Refuse — a training run is live hereNothing preempts it: not a higher-priority pool, not a backlog, not another queued run. The hold is re-checked every tick and releases itself the moment the run ends, however it ends.
  2. Refuse — this pool still has work in flightOr a co-resident pool sharing the same loaded model does. Work already accepted keeps the binding alive until it finishes.
  3. Act — a strictly higher-priority pool has a backlogThe one exception to the branch above, and the only branch in the whole list that interrupts a busy machine. It needs a strictly higher priority, a real backlog, and enough time since the last move.
  4. Refuse — an operator pinned this machineAutomatic switching was turned off on purpose. The platform does not second-guess that, and reads it as “cannot be pulled away either”, not merely “will not move itself”.
  5. Refuse — a batch reserved itA cooperative hold, released by whoever set it, so a long batch is not evicted halfway through by a rotation it has no way to see coming.
  6. Refuse — it has not been idle long enoughUnder five minutes of silence is not idleness. It is a gap between requests.
  7. Refuse — the engine reports active inferenceDirect traffic bypasses the platform's counters entirely, so the engine itself is asked before anything is torn down.
  8. Act — another pool has pending workRelease this one, bind that one, highest priority first. This is the ordinary move, and it only happens once every branch above has declined.
  9. Refuse — the loop will not start training by itselfReaching the bottom means the machine found nothing to serve anywhere. It still does not tear down its engine. Starting a training run is a separate decision, taken by a separate gate with eleven checks of its own — the last branch is a handover, not an action.
Order is the design Seven of the nine branches end in “stay”. Only two move anything, and the one that can interrupt work in flight is the narrowest of the nine. A scheduler that moves eagerly looks efficient on a utilisation graph and loses work in production — every eager move is a request that was already accepted and now has nowhere to land.
The drain gate

Eleven ways to refuse a training run.

Tearing down a live inference engine to start a training run is the most destructive thing this loop can do. It is the only move that ends a service in order to start a job, and it is gated accordingly: eleven separate checks run before a drain begins, and any single one of them ends it. Every refusal is recorded with its reason, so “why is this machine not training?” is a question with an answer rather than a hypothesis.

DELIBERATE HOLD OPERATOR PIN · BATCH RESERVATION WORK IN FLIGHT ENGINE SERVING · WORK QUEUED TIMING QUIET WINDOW NOT ELAPSED UNKNOWABLE STATE THREE OF THE ELEVEN THE DRAIN GATE ELEVEN CHECKS ALL MUST PASS ANY ONE FAILS ALL CLEAR REFUSED NOTHING IS TORN DOWN DRAIN, THEN TRAIN THE WAY BACK IS WRITTEN FIRST
Eight of the eleven checks ask whether the machine is busy or held. The three marked here ask something else entirely — and they refuse in exactly the same way.

The ordinary checks are the ones you would write yourself: an operator pin, a batch reservation, an engine that is still answering, work still queued for this machine, a quiet window that has not elapsed, no pool bound to drain in the first place. They all describe a machine that is doing something.

The other three describe a machine whose condition could not be established. A busy signal that could not be read. Two independent views of demand that disagree with each other. An inability to write down what would have to be put back afterwards. None of these says the machine is busy — each says the machine is unreadable, and all three resolve as refuse. Uncertainty is never read as permission. That is the difference between a scheduler that is wrong occasionally and one that is wrong expensively: the failure mode of an unread signal is a run that does not start, not a service that disappears.

RECORD THE WAY BACK ENGINE STATE CAPTURED WRITTEN DRAIN ENGINE STOPPED STARTED TRAIN RUN EXECUTES RUN ENDS RESTORE ENGINE PUT BACK IF IT CANNOT BE WRITTEN NOT STARTED NO WAY BACK, NO DRAIN
The order matters more than the steps. The way back is written down before anything is torn down, so the sequence has no point at which the machine is both stopped and unrecoverable.
Nothing is torn down before the way back is written down The engine's configuration is captured first; if that record cannot be written, the drain does not begin and the machine simply keeps serving. A machine that cannot be restored is not allowed to be repurposed. When the run ends — successfully or not — the engine is rebuilt from that record and the machine returns to the rotation.
What it changes

The alternative is a box per workload.

Rotation is not a utilisation trick. It is what lets a small fleet carry several kinds of work without each kind needing its own idle machine.

Without rotation

One box per workload, each of them idle most of the day. Training waits for a free machine or gets one of its own. Someone decides what runs where, and finds out too late when they got it wrong. Urgent work queues behind whatever happened to start first.

Paid for by the hour, either way

With it

One box moving between workloads as they arrive. Training uses the gaps in serving and hands the machine back. The machine decides, and records the reason every time it declined. A higher-priority backlog can move it mid-stream, at most once a minute. Only a strictly higher priority does that; an equal or lower one never interrupts, and never before the dwell floor has passed.

Same hourly rate, more of it worked
This is not a warm pool Machines here are billed by the hour and expected to be worked, not held idle to shave a cold start. The rotation loop optimises for a busy box; when a box cannot find anything to be busy with, that is a different mechanism's problem — see reaper chains, which stop paying for it.
The series Pool rotation is part four of six. All six mechanisms →

Run cheap GPUs without the bill shock.

Request early access →