An ops agent on shift

Since early July, SplitKit's production has been watched by a reasoning agent: an always-on process, running on hardware I control, that wakes on a cadence, reads the same telemetry a human operator would, and reasons about whether the race is actually performing. When it concludes something is wrong, it pages me. The rest of the time it writes down why it is confident nothing is. It is the most interesting piece of the operations stack, and it was deliberately the last piece built.

That ordering is the entire lesson of this entry. You need the foundation for the agent to work: an agent reasoning over no signals is a hallucination engine, and an agent reasoning over good signals is a colleague. The stage map in From idea to production with AI in the loop ends at operate. What follows is that stage opened up, in the order the layers actually went in.

1.0The floor goes in first

SplitKit is built and run by one person, no team, and the standard move for a solo build is to treat operations as a later problem: ship, then add observability when something forces the issue. The schedule here ran the other way. Durable logging went into the written architecture principles before any application code existed, on the reasoning that a log you did not keep is an incident you cannot explain. By week four of the project the rest of the classical floor was in place: telemetry draining from every surface into one queryable place, uptime monitors on the public routes, heartbeat checks on every scheduled job, and threshold alerts wired into a team channel, so a stalled cron or a failing route announces itself the boring way. That layer runs on BetterStack, and nothing about it is novel. It is the floor every ops organisation stands on, and being an organisation of one does not exempt you from pouring it.

What that floor cannot do matters just as much, because it defines the second layer's job. A threshold does one thing well: notice a number crossing a line. It is structurally blind to anything that takes two numbers and a story. "The results feed looks fresh but the poller has gone quiet" is not a threshold; it is an inference across signals. So is "these counts are technically valid but make no sense at this point in a race." Every ops team has a person who catches those. The threshold layer was never going to.

2.0A throwaway proves the gap

I learned that concretely on the platform's first production weekend, which was watched by a bash script written for exactly that weekend and nothing else.¹ It knew one event, it checked staleness and polling cadence against fixed limits, and its entire model of normal was whatever I guessed while writing it.

It earned its keep mid-race. Every dashboard was green: routes up, logs flowing, not a threshold alarm anywhere in the general monitoring. The watcher flagged the one thing none of that measured: the published results the live page polls had stopped getting fresher, 160 seconds old against its 150-second limit, which meant the pipeline between the timing system and the spectators' screens had quietly stalled and the live view was about to freeze. Everything upstream reported healthy, because every component, considered alone, was healthy. The question that mattered, is the data spectators see actually moving, belonged to no single component, so no standard monitor was asking it.

That one catch justified the whole category of watcher. It also made the ceiling of the approach vivid: the stall was caught only because I had guessed that exact failure in advance and typed the limit in. Everything the script could ever catch was a line being crossed, and everything else I worried about that morning was a pattern. Within days it was superseded by the reasoning agent, and it was retired the same day its successor shipped. A throwaway exists to be thrown away; this one also existed to prove exactly where thresholds stop.

3.0Briefed like a hire, not programmed like a rules engine

The agent's entire specification is a 12.8 kilobyte operating manual, and it reads like the briefing you would hand a new hire on the ops rotation, because that is what it is. The role framing comes first: you are the layer above the threshold monitors, and your whole job is judgment. Then a plain-language map of the live data pipeline and what the agent sees each time it wakes. Then a playbook of five failure modes, named and described by shape so they can be recognised in new clothing: an event finalising before the race is over, a mid-race revert to the timing platform's test mode, polling that has quietly stalled, masked test data leaking through as if it were live, and a cancelled contest borrowing a sibling's counts. Then the baselines, including the counterintuitive ones a new hire would trip over: certain counters are deliberately not additive, so a small mismatch between them is normal, and raising it would be noise.

Each wake, the agent walks a six-question reasoning agenda the manual frames as doing rounds, the way a human on shift walks the floor. It answers in a strict JSON contract: an assessment; any alerts, each with a severity, its reasoning, the signals behind it, and a suggested fix; when it wants to wake next; whether it needs a deeper look; and a findings note addressed to its own future self, because the next tick starts cold and the handoff note is the only memory it gets.

One rule in that manual outranks the others: the agent is forbidden from duplicating anything the threshold layer already alarms on. If a coded monitor pages on it, the agent stays silent about it. Its whole value is connecting dots that fixed rules structurally cannot, and every duplicated alarm would spend the trust that makes its pages worth waking for.

4.0Cheap ticks, gated depth

An always-on reasoning layer sounds expensive, and built naively it would be. Every wake runs cheap by default: the manual plus a pre-collected telemetry snapshot, no tools, a single pass. Only when a tick cannot explain something it is seeing, or is about to page, does the question get an upgrade: a re-run on a deeper model with read-only access to the project wiki and the code repositories, bounded in how many turns it may take. Calm ticks stay near-free; anomalies pay for depth, and only anomalies. The cadence stretches and contracts the same way, and the agent proposes its own next wake time inside limits the wrapper enforces rather than trusts.²

5.0Paging is a promise

A watcher that pages wrongly is worse than no watcher, because it trains the human to stop answering. So paging carries a three-part test, and all three parts must hold: the problem is happening now, it is material, and it is corroborated by at least two independent signals. When they do hold, the finding lands as a Slack message to me, the one human on the rotation, with the reasoning and the corroborating signals attached: what wakes me is an argument, not a number. A frozen pipeline qualifies because two signals agree: the freshness timestamp gone stale and the polling telemetry gone silent at the same time. A one-signal hunch never pages; it is recorded as an FYI in the findings file. And nothing already raised in a previous handoff gets paged again unless it has become materially worse.

The findings file is an append-only markdown log, one entry per tick: timestamp, a state summary, and the bullets behind the agent's confidence. A quiet race weekend runs to tens of thousands of lines of "nothing changed, and here is the evidence," which is precisely what a watcher's log should look like. Judgment flows back the other way through a separate acknowledgements file, and closing a finding there costs me something: each closure carries a root-cause explanation and an explicit re-raise condition. One real acknowledgement closes a one-participant count discrepancy with the instruction to never mention it again unless the gap widens, and the agent has honoured that condition since, so the finding stays conditionally closed rather than forgotten. Another documents a bug in the agent's own snapshot-builder, which the agent caught and wrote up in its own findings format. The watcher watched itself, and its bug report reads like all its others.

6.0Fail towards the pager

The last layer of the design assumes the watcher itself will fail, and arranges for every one of its failures to end in a page rather than in silence. Each tick runs its steps in a deliberate order, and the order encodes the principle. The first step is a credentials check, and it contains my favourite decision in the whole system: if the read-only credentials are broken, the script deliberately does not ping its own alive-heartbeat. It could truthfully report that the process is running. It refuses to, because a blind watcher running is worse than a dead one, and the withheld heartbeat trips a dead-man's switch that pages a human. Discovery of watchable events comes next and costs no model tokens at all. The live-reasoning heartbeat is pinged only after a reasoning pass succeeds, so a failed model invocation trips the switch too. And production keeps its own witness: the platform's API pings a heartbeat that says a live event exists right now, and each tick cross-checks that witness against what it discovered. A gap between the two must persist before anyone is paged, and any successful discovery clears it, so a transient blip never wakes anyone before dawn on a race morning.

There are four dead-man's heartbeats in all, and each silence means exactly one thing.³ A bare machine pulse runs from boot with no credentials and no AI involved, so its silence can only mean the hardware is down. The per-tick heartbeat means the agent is alive. The post-reasoning heartbeat means judgment is actually being exercised while an event is live. And the witness heartbeat comes from production itself, closing the one blind spot the other three share: an agent healthy in every respect except that it has failed to notice a real live event. All four live on Healthchecks.io, deliberately a different provider from the BetterStack stack watching production, so the watcher's liveness never shares a failure domain with the thing it watches. Even the switch is queried with an explicitly read-only key.

7.0If you're doing this yourself

The build order, with the gate or the rule that goes with each step:

1. Pour the floor first. Durable logs written in as an architectural principle, telemetry drained from every surface, uptime monitors, heartbeats on every scheduled job, threshold alerts into a channel humans actually read. All of it before anything clever. Gate: if the model API vanished tomorrow, would you still hear about tonight's outage?
2. Add reasoning only where thresholds structurally cannot go. Rule: the agent may never duplicate anything the threshold layer already alarms on. Write the prohibition into its operating manual and mean it.
3. Brief the agent like a hire. Failure modes described by shape, baselines including the counterintuitive ones, a reasoning agenda to walk each wake. Gate: could a competent human on their first shift operate from the same document?
4. Make paging expensive for the watcher. Rule: current, material, and corroborated by at least two independent signals, or it does not page. One-signal hunches go to a file, not a phone.
5. Close findings with conditions, not deletions. Rule: every acknowledgement carries a root cause and an explicit re-raise condition the agent must honour.
6. Fail towards the pager. Order each tick so any failure of the watcher's own senses withholds its heartbeat, and host the dead-man's switch with a provider that shares nothing with your production monitoring. Gate: when the watcher goes blind, does someone get paged about the silence, or is it just silent?

The agent is the layer people ask about. The floor is the layer that makes it worth having. Build them in that order.

Entry 05 · The watcher described here was on shift while this was written