All You Need Is Prod

AI for AI quality

Evals are only as good as the environment they run in. Production is the only environment that is always accurate: it’s where failure modes show up, and it should be where they get fixed. So instead of building synthetic environments, treat a read-only mirror of production as the fixture for experiments.

What a Slack Agent Taught Us

We recently shipped a Slack app where people discuss and analyze an AI agent’s traces, and collect them into datasets, by talking to another agent. It did several things at once: it lowered the barrier to grokking traces, it queried traces at scale, and it handled complex questions in natural language. Adoption was quick. Connecting it to the codebase went further: it could compare how an AI feature is used with how it’s implemented and configured.

The bottleneck wasn’t the analysis, it was the friction to start one. What if testing a quality improvement was just as easy?

Bring the Same Experience to Experiments

You: Based on today’s trace review, latency is up because the agent calls tool xyz too often.
Agent: Plan: test N variants of xyz’s metadata on last week’s traces where it was called 5+ times in a turn. I’ll measure tool calls per turn, turn latency and answer quality. OK?
You: Go!
[…]
Agent: Variant B: −10% xyz calls, −5% latency, quality unchanged.

The person is left with two decisions: approve the plan, and decide what to do with the results.

But building a cohesive synthetic environment, plus the synthetic user conversations to go with it, is slow, barely representative of real users, and bound to drift. How do you fix failure modes seen in production with quick experiments?

Make Production the Fixture

I see four pieces, each useful on its own.

Mirrors. Spin up a read-only mirror of production, scoped to the accounts a dataset was drawn from.¹ If traces and datasets are tagged by account, the first set of parameters comes for free, whether it’s a real account, a test one or a synthetic one. That alone runs the latency experiment above. Others are converging on the same idea: DoorDash’s simulation flywheel, or Shopify reflecting production in its evals.

Overlays. Agents increasingly make changes on users’ behalf, so replays must handle writes. Each replayed example gets its own writable overlay on top of the mirror: changes stay isolated, and the agent can read its own writes. The lifecycle is a test fixture’s: prepare, run, collect results, discard. This extends experiments to agents that draft documents, reassign owners, and so on.

History. Present-day state distorts replays. By the time you replay a trace, the environment has almost certainly changed. Applying the original conversation’s writes is necessary but not sufficient: those writes did happen, after all. The goal is a reasonable approximation of past state. If you already have the perfect time-machine, lucky you!

Experiments are only as accurate as the replay environment is close to the historical one. Data added since may slow down retrieval tools, and a document the user asked to draft may since have been polished and approved, making the replay moot.

Replays. Finally, a framework to drive multi-turn replays: send the user’s follow-up messages, stop when they no longer apply to what the new agent did, or switch to simulated user messages from there, as Sierra does. That turns a replayed turn into a replayed task.²

Each Piece Pays for Itself

Mirrors alone cover any read-only change: prompts, tool metadata, model choice. Overlays add agents that write. History makes the results trustworthy. Replays stretch all of it to whole conversations. None of it needs a synthetic world that someone has to keep in sync with reality.

All you need is prod.