AI in the product itself, without setting money on fire

Every other entry in this collection describes AI on the builder's side of the counter: planning the product, writing the code, reviewing the pull requests, watching production. This one crosses the counter. It is about putting AI inside the product itself, where a customer pays for what the model writes and takes it away, and about doing that safely and at a cost you can survive.

The concrete case: there is a moment in SplitKit, the race-day platform I designed, built, and operate (one person, no team), where a runner pays real money and part of what arrives was written by a model: a multi-page analysis of their race, built from their actual split times, addressed to them in the second person, meant to be saved, printed, and handed to a coach. The purchase unlocks several server-generated artefacts, two of them model-written; this entry follows the long-form report, because it carries the most inference cost per sale. It is a deliverable: a model call inside the purchase flow, not a chat window or a copilot feature bolted onto the page.

That placement changes the accounting. In a free feature, a wasted model call is a rounding error. Inside a paid path, every token is cost of goods, and a call that produces nothing the customer keeps is money spent on nothing. The prompt took an afternoon to write. The contract around the call, when it runs, what counts as success, what happens when it fails, took the real engineering, and it is the part that transfers to any system that puts inference next to revenue.

1.0Generate once, at payment time

Ask an agent to "show the buyer an AI-written report" and the default shape comes back quickly: when the reader opens the page, call the model, show a spinner, cache the result. It demos beautifully. In production it has two structural problems no cache fixes. The render path now depends on a third-party API at the exact moment a customer is looking at something they paid for, and cost now scales with traffic instead of with purchases: refreshes, crawlers, and a coach opening the link twice all become billable events.

The rule that shipped instead: the model runs exactly once per purchase, at the moment the payment webhook confirms the charge, inside the API layer. The output persists onto the participant's record and is served as static data through the CDN. The route that renders the report contains no model client at all; it reads a body that already exists. The render budget at the edge was half a second, and a synchronous model call on that path would have made the budget fiction.

The same rule cuts the other way too. Generation does not run early. Producing a report at finish-line time for every runner, when only some of them ever buy, would spend inference on documents nobody purchased. The call happens after the money event and before the render. That window is the whole design.

2.0The webhook arrives more than once

Payment webhooks are delivered at least once, which means sometimes more than once. Redelivery is documented, normal behaviour, and a duplicate delivery that re-runs the generator double-bills the inference for a single sale. So the handler is idempotent on the artefact itself: before invoking the model, it checks whether the report body already exists on the participant's record, and if it does, the delivery is a no-op. Not a dedupe table of webhook IDs; a check against the output. "Did I already do the work?" is answered by looking at the work.

The subtler rule is what counts as success. The model call is not complete when the API responds; it is complete when the body and its metadata are written durably to the participant record.¹ A successful response that fails to persist is a failure, and the retry policy follows from the economics: retry the write, not the call. Re-invoking the model pays a second time and, because sampling is non-deterministic, produces a different document. Retrying the write costs nothing and preserves the exact words the first call bought.

3.0Freeze the output, not the prompt

Prompts get tuned continuously; artefacts are permanent. Those two facts collide unless you pick a rule, and the rule here is strict: there is no cache invalidation on prompt changes. A coach opening a six-month-old report sees exactly the document the runner saw on race day. Tuning the prompt affects new purchases only; everything already generated stays frozen.

This inverts the usual instinct about caches, where a stale entry is a bug. Here the staleness is the contract. The output is the only stable thing in the system: the prompt evolves, model versions roll over, and the same prompt run twice produces different sentences. When the artefact is something a customer paid for and might cite, the generated words are the record, and the record does not change because the factory retooled.

4.0Choose the loud failure

If a paying customer opens their report and the body is missing, the route returns a 404.² It does not fall back to generating on demand. The lazy-generation fallback is the seductive option because it looks like resilience: the customer still gets a document and nobody files a ticket. What it actually does is convert a generator-side bug into an invisible recurring spend. Every affected reader triggers a fresh model call, the money leaves quietly, and the bug stays hidden precisely because its symptom has been papered over. A 404 is countable, alertable, and embarrassing enough to get fixed.

Regeneration exists, and it is user-initiated only. The concrete case is language: a report generated in one language, read later in another. The page renders the original with a small note naming its language, and producing a new version is an explicit action the reader takes, never something the render path decides on its own. No code path in the system spends inference without either a payment event or a deliberate human request behind it.

5.0A hand-written document is the regression spec

The report's system prompt carries voice rules that read like an editor's standards, because that is what they are. Second person throughout. Every observation cites a specific number from the splits or the placement data; value-judgment adjectives are banned, so the output says "the slowest segment of the day" with the pace attached rather than "a tough stretch". Motivational platitudes are banned outright. Training advice may only follow from a pattern actually present in the splits, and when the data points at no clear lever, the prompt instructs the model to say so rather than invent one.

Rules like that need enforcement, and the enforcement here is a document rather than a metric. The demo event's report was written and tuned by hand to those rules, and it serves as the reference output: when production generations drift from that voice, the drift is treated as a regression and the fix lands in the prompt. For one artefact at modest volume, this beats a scoring harness. The reference is cheap to maintain and legible to a human in thirty seconds. It also catches tone, the class of failure that matters most for a paid document and the one numeric evals handle worst.

6.0Watch the money before there is money

The platform's founding cost gate, covered in "From idea to production with AI in the loop", asks what the system costs when nothing is running. Inference in a paid path adds the counterpart question: what does each sale cost while things are running? Answering it needed observability that most projects defer until the bills hurt, so it went in before launch: a single admin surface pulling real spend from every provider with a billing API. AWS through Cost Explorer, Twilio usage records, Anthropic's admin cost report, Resend, and Vercel's FOCUS-format billing export.

Five adapters, one shape: is it configured, a month-to-date total, a daily series, and a typed last-error. Every adapter soft-fails to null, so a missing key or a provider hiccup renders as a dash with a specific diagnostic instead of a thrown error or a blank page. That property earned its keep immediately: during the first deployment, the typed diagnostics surfaced every misconfiguration on the page itself, without a single round-trip through the logs.

Building the adapters was also a tour of how lightly providers exercise their own billing APIs.³ One provider's cost report silently truncates a thirty-day query to its first week by default, and the same API reports amounts in lowest currency units where the documentation reads like dollars, which made a tile display one hundred times the real spend until it was reconciled against the provider's console. Another files its monthly subscription fee in the same envelope as metered usage, quietly inflating any naive burn calculation. The rule that came out of that week: a billing adapter is unverified until its number matches the provider's own console.

7.0If you're doing this yourself

The checklist, each item with its gate:

1. Put the call at the money event. Generation belongs where payment is confirmed, not where the page renders. Rule: the render path holds no model client.
2. Assume every webhook arrives twice. Make the handler idempotent on the output itself, not on delivery bookkeeping. Gate: what does a duplicate delivery cost you?
3. Define success as persisted. A model response that never lands in storage is a failure. Rule: retry the write, not the call.
4. Freeze outputs against prompt changes. Prompt tuning applies to new purchases only. Gate: what does the customer who bought six months ago see after this week's prompt tune?
5. Choose the loud failure. Lazy-generation fallbacks make the system quieter while it spends. Gate: when the generator breaks, does the system get louder or quieter?
6. Write the reference output by hand. Rule: treat production drift from the hand-tuned document as a prompt regression, and fix it in the prompt.
7. Wire cost observability before revenue. Soft-fail every adapter to null with a typed diagnostic. Rule: no billing adapter is trusted until its number matches the provider's own console.

Nothing here required novel research. It is metering, idempotency, immutability, and reconciliation, the ordinary disciplines of any system that touches money, applied to a dependency that writes prose and bills by the token.

Entry 06 · The render route still holds no model client