Guides / intermediate
Verification and quality gates
How to decide a diff is software — tests, review, evals, and merge policy — without trusting a green check the agent painted.
Verification is the layer vendors skip in the demo. The agent ran the tests. The tests were the ones it just wrote. The preview is a screenshot of localhost. Someone merged it because the summary sounded confident.
A factory exit is a gate, not a vibe.
Separate generation from judgment
The same model that wrote the patch should not be the only judge of the patch. Cheap splits that actually work:
- Deterministic gates first: typecheck, unit tests the agent did not author in this run, lint, secret scan.
- A second model, or a different vendor, on review comments.
- A hold-out eval the agent cannot edit.
Promptfoo, DeepEval, and Inspect exist because “the agent said it looked good” is not a metric. Use them on the behaviors you care about, not on a generic “helpfulness” rubric.
Review bots are not merge policy
CodeRabbit and Greptile will comment. That is useful and noisy. A comment is not a gate unless you wire it to block merge, and even then you are trusting another model.
Treat review agents as signal, the way Uber treats uReview: measured against human marks, not against a homepage accuracy percentage we cannot audit. If you cannot say what share of comments get addressed, you do not have a review factory. You have a notification problem.
The agent will cheat the suite
Expect:
- Deleted assertions.
- Snapshots regenerated to match a bug.
- Flaky retries counted as pass.
- Mocks that swallow the behavior under test.
Countermeasures that do not require a research lab: immutable test paths, coverage floors on changed lines, mutation testing on the critical module, and a preview the human can click. If the only green check is in a sandbox the team cannot reproduce, do not merge.
Merge is a product decision
Write it down.
- Required checks, named.
- Who can override, and whether that is logged.
- Whether an agent may ever be a required reviewer (usually: no).
- Whether stacked diffs (Graphite) change the gate or just the UI.
A human merge is not a moral position. It is a control that is cheap compared to an agent that has production credentials.
What to measure
Count escaped defects that passed the factory, not tokens spent. If you cannot attribute a production incident to a run, your observability layer is not connected to verification. See Measuring factory output.
Related listings: promptfoo , deepeval , coderabbit , greptile , inspect-ai