AI Content Platform10K daily users · 12 models · 35% lower cost

A content SaaS had scaled their MVP from 200 users to 4,000 on a LangChain-JS prototype. It had carried them further than it should have and was now the source of nearly every production incident: surprise provider outages, opaque failure modes, unbounded costs, and a debug story that consisted of printing messages and crossing fingers.

They wanted to keep shipping features. We needed to replace the guts without anyone noticing.

• Zero user-visible downtime during the swap. Feature velocity could not drop for more than a sprint.
• Cost per generated document had to come down — the growth curve was going to take them through $250K/month in OpenAI fees unless something changed.
• Every model call had to be replayable, evaluable, and auditable. “It worked in dev” was not acceptable.
• Tenant isolation at the data layer, not the application layer — enterprise customers were asking.
• Provider failover in seconds, not retries-and-prayers.

At the center: a small orchestration runtime, maybe 1,200 lines of TypeScript, that models a generation as a directed graph of typed steps. Each step is a pure function of its inputs plus a retry policy, a budget, and a fallback provider. Nothing clever. Everything inspectable.

Around it: a provider adapter layer (OpenAI, Anthropic, Gemini, a local Llama for the cheap-and-fast tier), a cache keyed on normalized prompts, and a structured eval harness that runs a sample of traffic against a golden set on every deploy. Regressions fail CI.

Tenant isolation lives in Postgres row-level security, not application code. Metered billing reads directly from the same audit log that feeds the eval harness — one source of truth, used three ways.

• Streaming fallback. When the primary provider dies mid-stream, failing over without the user seeing a broken paragraph required buffering the last 200 tokens and surgically splicing.
• Cost modeling. OpenAI's token accounting does not always match your tiktoken estimate. Billing on our measurements rather than theirs took a month of reconciliation.
• The “obviously cached, actually not” bug. Two prompts that looked identical to a human hashed differently because of invisible unicode. Normalization is load-bearing.

10K+ daily actives, twelve production models across four providers, 35% lower inference cost per document than launch month despite the user base tripling. P99 streaming latency is 2.1s end-to-end. The eval harness catches a real regression roughly once a month before it ships. Exactly one provider has had a multi-hour outage; users saw nothing.

I'd lean harder on structured outputs from day one. We bolted JSON-schema validation on in month four and it immediately deleted a class of downstream parsing bugs we had been chasing for weeks. I'd also build the eval harness before the orchestration runtime, not alongside it — evals change how you design the pipeline, not just how you test it.