Guides / advanced
Measuring factory output
What to count when agents write the diffs — and which dashboards are just token spend with a nicer chart.
The easiest factory metric is tokens. It is also the least related to whether you shipped software.
A factory produces merged, verified changes that survive contact with users. Everything else is a leading indicator at best and a vanity chart at worst.
Count the loop, not the chat
Useful questions, in order:
- How many runs reached a mergeable PR?
- How many of those PRs merged?
- How many merged PRs were reverted or hotfixed within a week?
- How long from signal to merge, and where the time went (queue, agent, review, CI)?
- What it cost in inference and in human review minutes?
If you only have (5) and a thumbs-up on a chat transcript, you are measuring a copilot, not a factory.
Shopify published River numbers from a table the agent writes itself: sessions, channels, merged PRs. That is the right shape even if you will never see those volumes. Stripe’s Sessions 2026 keynote separated “minion PRs to production” from “PRs created with AI assistance.” Steal that distinction. Assisted and autonomous are different factories.
Do not invent a productivity percentage
We will not give you a “30% faster” target. The public numbers that survive a source check are almost always counts (PRs, sessions, comments marked useful), not productivity multipliers. When a vendor homepage offers a multiplier, leave it on the homepage.
Uber’s uReview post is usable because it says what was measured: share of diffs analyzed, share of comments marked useful, share addressed. You can copy the method without copying the numbers.
Trace the run
Langfuse, LangSmith, and Braintrust will store prompts and spans. That is necessary and insufficient. Tie the trace to:
- The issue or spec that started the run.
- The sandbox ID and the preview URL.
- The PR and the merge commit.
- The CI jobs and which ones the agent reran.
A trace that dies at “model said done” cannot explain a production incident.
Eval in the loop, not on a poster
Offline evals drift. Run a small hold-out on every harness change: the same ten tickets, the same pass conditions, a second model as judge only after deterministic checks. Factory sells Agent Readiness dashboards for this; you can also keep a spreadsheet. The sin is shipping a new prompt and calling the week’s merge count a success.
Cost is a budget, not a flex
StrongDM’s factory page treats “$1,000/day/engineer in tokens” as a koan, not a benchmark. We do not repeat it as a target. Spend what the gates are worth. If review time went up more than cycle time went down, the factory is a tax.
Put the dashboard next to the methodology, not next to the sales deck.
Related listings: langfuse , langsmith , factory , braintrust