← Machinery Machinery · 02

How does it know?
One writer per question.

The previous page describes five watchdogs that stop paying for a machine which has stopped working. This one answers the question a sceptic asks immediately afterwards: how does any of them know? A watchdog is only as good as the field it reads — so every fact about a machine has exactly one component allowed to write it, and a written-down answer to which machines that component actually covers.

The rule

Two writers is survivable. Partial coverage is not.

The contract in one line One question · one record · one writer · and a declared set of machines that writer covers.

Two components answering the same question is a nuisance you can survive. You can compare the two answers, notice they disagree, and pick. A record whose writer covers only part of the fleet, read by automation that assumes it covers all of it, is not survivable — because nothing about that read looks wrong. It returns a value. The value is well-formed. It is simply about a machine the writer was never responsible for, and no amount of care at the call site can tell.

That distinction is the whole of this page. It is also what makes the watchdogs on the previous page worth trusting: they act on money, they act unattended, and they act on a field. The field has to be honest about its own reach.

Who is allowed to answer

Three parties can speak. None of them speaks for another.

Three parties have something to say about a machine: the machine itself, the control plane that decides what to do with it, and the provisioner that rented it and is the only thing able to destroy it. Each writes a different set of facts, and no fact has two writers.

The machine's side of that is deliberately narrow. It reports through one channel with a fixed vocabulary — one of 185 named conditions, each of which maps to exactly one of nine instructions to automation. It cannot report a sentence. An engine that dies at three in the morning does not get to invent a new word for how it died: it emits a condition the control plane already knows how to route, or it emits nothing and the silence itself becomes the signal. Free-form status reporting is not a supported input.

THE MACHINE REPORTS ITSELF ONE CHANNEL FIXED VOCABULARY CONTROL PLANE INTERPRETS WRITES FLEET-WIDE FACTS SET ON EVERY MACHINE WHAT AUTOMATION READS NO SECOND CHANNEL REFUSED FREE-FORM STATUS THE PROVISIONER RENTS · DESTROYS WRITES RENTAL FACTS ONLY MACHINES IT RENTED BLANK EVERYWHERE ELSE
The two record sets are disjoint, and the machine has exactly one way to speak. The fleet-wide facts are set on every machine, which is what makes them safe to act on; the rental facts describe only machines the platform rented, which is what makes them unsafe to act on without checking.
The ledger

Nine questions, nine owners, nine declared coverages.

Nine questions get asked about a machine often enough to be worth storing the answer. Read the last column as the audit: almost everything that goes wrong here goes wrong there, not in the writer and not in the number of questions.

AskedThe questionWho is allowed to answerWhich machines it covers
Act What should automation do about this machine, right now? The control plane, from what the machine reported Every machine, always set — the only answer a watchdog is allowed to act on
Condition Which named condition is it in, out of the catalogue? The control plane, same path Every machine — drives the dashboard and the audit trail, not the action
Narrative What is happening right now, in words written for a person? Whichever controller is doing the work Every machine — display only, never routing
Progress How far through the current step is it? The same controller Every machine, left blank where a step has no meaningful fraction
Cause Why did it fail, in a category automation can switch on? The controller that observed the failure Failures only — absent while things are going well
Rental Where is this rented machine in its provider's lifecycle? The provisioner Only machines the platform rented itself — blank on everything else, and blank means “not ours to answer”, never “idle”
Intent What state do we want it to be in — as opposed to what it is in? The provisioner Only machines the platform rented itself
Verbatim What did the provider itself last say about it? The provisioner Only machines the platform rented itself
Since When did the current condition begin? Whoever performed the transition Every machine — every grace window and every timeout is measured from here
Why two of these are not the same fact A machine's position in a rental lifecycle and an instruction to automation are different questions, and neither is a substitute for the other. They get confused because both can be described in English as “the status of the machine” — which is exactly why facts here are filed under the question they answer rather than under a name someone chose once.
Prose is not control flow The narrative answer exists for people. It is overwritten on every progress event, and nothing routes on it. Retry and alert decisions switch on the failure category instead — a small stable set that every consumer is required to default-case. A regular expression over an error message is not a control flow; it is a bet that nobody will ever improve the wording.
The failure chain

Every step in this trap is a reasonable decision.

A fact is written by one component and read by another. Nobody writes down which machines the writer covers, because on the day it was built the writer covered every machine that existed. Later the fleet grows a second kind of machine — one the platform did not rent, or one that reports through a different path — and the writer, correctly, does not touch those machines. The fact is now sparse. Everything still works, because the readers that matter happen only to look at machines the writer covers.

Nothing has failed yet. What has happened is that a fact acquired a coverage that exists only in the original author's head. From there, three outcomes become possible, and they compound.

UNDECLARED COVERAGE NOBODY WROTE DOWN WHO IT COVERS PARTIAL IN PRACTICE WRITER COVERS SOME MACHINES READ AS TOTAL CALLER ASSUMES EVERY MACHINE FROZEN VALUE TAKEN AS CURRENT FACT ARMED TIMER NOTHING CAN CLEAR IT DELETED LOOKED UNUSED TO BOTH
The hinge is the third box, not the second. Partial coverage is a fine engineering decision; reading it as total is the mistake, and the read looks identical either way.

The first two are unpleasant but recoverable. A frozen value is a label nobody has refreshed since it was written, and a caller that treats it as current decides about the machine as it was, not as it is. A timer armed behind that value is worse: the path that would disarm it filters on the same frozen field, so no amount of observable activity on the machine can turn it off. Both leave evidence.

The third one leaves no evidence at all, and it is the reason this page exists.

SERVICE A WRITES THE FACT SEARCHES ITS OWN CODE FINDS NO READER WRITES ONE FACT WRITTEN HERE READ OVER THERE READS SERVICE B READS THE FACT SEARCHES ITS OWN CODE FINDS NO WRITER NEITHER SIDE SEES A USE TEST SUITE A STILL GREEN DELETED AS LEGACY TEST SUITE B STILL GREEN THE OTHER SERVICE STOPS
Both searches are correct. Both conclusions are wrong. Each side's tests exercise its own half against a database it set up itself, so the removal is green on the way in and only observable in production.

A fact that one service writes and another service reads looks unused from either side alone. Search the writing service for readers: there are none, because the reader lives in the other codebase. Search the reading service for writers: there are none, for the mirror-image reason. Both searches return the truthful answer to the question that was asked, and the question was the wrong one. Remove the fact and the other service stops — not degrades, stops — and it does so without a single failing test, because neither suite ever had both halves in front of it.

This is not a discipline problem Nobody in that sequence was careless. The information needed to make the right call did not exist in either place, so no amount of review would have surfaced it. The fix is not more care — it is moving the dependency out of people's heads and into something a build can check.
The guard live

A contract nobody checks is a comment.

The facts one service depends on inside the other's schema are declared in a list that both sides verify on every change. Five checks, in the order they run — three that act, two that refuse.

  1. Act — declare the coverage, next to the fact Every stored fact carries a written statement of which machines its writer is responsible for. Not in a design document, not in someone's memory — attached to the fact, where a reader looking at it will see it.
  2. Act — derive the dependency list from code that actually runs The list of facts one service needs from the other is extracted by parsing the queries that service really issues, not by searching for a name as text. A text search cannot distinguish a stored field from an identically named key inside a document stored in another field, and at least one of these appears as both.
  3. Act — replay every schema change and assert the end state The full history of schema changes is replayed against a real database, and the result is compared against the declared list. Both services hold a copy of that list and the copies are diffed on every change, so they cannot drift apart quietly. A change that removes a declared fact fails before it merges.
  4. Refuse — a query that will not name the fields it reads A query that asks for everything hides which facts it depends on, which makes it invisible to the check above. That is not a style preference here: an unnameable dependency cannot be protected, so the query is rejected rather than exempted.
  5. Refuse — a deletion justified by one side's search Empty is not the same as dead. Sparse and dead are indistinguishable from the data alone, and some facts are empty most of the time because they describe conditions that last seconds. A fact is removed when its writer is known and gone, not when a search came back quiet.
Four ways to get this wrong

The same mistake, wearing four different hats.

Each of these is a reasonable-sounding step that quietly converts a fact into a guess.

Calling it legacy because you stopped reading it

Another service may not have stopped. One-sided evidence about a shared record is not evidence, and the cost of being wrong is a second system going dark for reasons that appear unrelated to the change.

Fix — find the writer and the reader, both

Reading a fact about machines you do not maintain

Outside its writer's coverage, the value is whatever a one-time import left behind, frozen from that moment on. Scope the query to the machines the writer is responsible for, or ask a different question.

Fix — scope the read to the declared coverage

Adding a status value to express progress

Progress has its own field, with its own range. Every value added to a routing enum silently changes the meaning of every consumer's default branch, including the ones you cannot see from your repository.

Fix — put progress in the progress field

Reading an empty field as a dead one

Some conditions are real and last seconds. A field that measures empty across the whole fleet may have a perfectly healthy writer describing something that is simply rarely observed at rest.

Fix — find the writer before concluding there is none
The series This is part two of six. All six mechanisms →

Run cheap GPUs without the bill shock.

Request early access →