
Part 3 of The React Native Software Factory, a series about tuning the setup around coding agents until you can leave it alone for a night.
In October, I was trying to stop comments from going stale in Simlock, a tool we build at Callstack that lets agents lease simulators and physical devices for testing.
The coding agents I use to develop it were changing the code while leaving behind comments that described how it used to work. I decided to give a separate reviewer agent the job of catching that.
I wanted to check that comments still explained the code accurately, including why it behaved unexpectedly.. We added a rule, including a required why: prefix, for the reviewer to enforce. On the next large pull request, it raised 155 findings marked as blockers, each of which had to be resolved before the change could proceed.
Another agent verified them, the implementation agent made the fixes, and the pull request went through five review rounds.
I'd added this reviewer to improve the code, and now I had another queue of work to manage. Even a correct finding needs someone to verify it, make a fix and check that fix. Reviewers can also miss defects, so the number of findings alone couldn't tell me how well the review was working.
That experience led me to rethink the whole review process: what a fresh agent should see, which questions it should investigate, and how to get from a reported problem to a finished change. How could I get a reviewer to challenge the writer's assumptions without keeping the two of them busy forever?
In this article, I’ll show how we give independent reviewers distinct jobs, verify their findings, and recheck fixes within a limited number of rounds. I’ll also explain how we replay past pull requests to investigate what the reviewers missed.
Start with an AI reviewer who hasn't written the code
In Simlock, an implementation agent starts from an approved GitHub issue. For a feature, the issue describes one task and links to the larger feature plan. The agent writes code and tests, then opens a draft pull request. The previous article covered how we prepare that task; this is where we check whether the code delivers it.
I want another agent to investigate the change without having followed the conversation that produced it. The writer has already settled on an approach and explained why it should work (we talked about it in the previous part of this series). Passing that explanation along gives the reviewer a ready-made interpretation before it has read the code.
So the reviewer receives the code, the requirements and the repository's rules, with the code fixed at a specific commit. It doesn't receive the implementation conversation, pull request description or commit messages. It has to work out how the implementation behaves from the source.
In an earlier version of this setup, we also withheld the specification from the code reviewer. It kept objecting to choices we'd deliberately made during planning, so we added the specification to its inputs. The reviewer needed to know what we’d agreed to build while investigating the implementation independently.
-1920w.webp)
Give the reviewers different jobs
Checking whether we built the requested feature is different from finding a way to break its implementation. Neither check necessarily catches outdated documentation: we might remove a command while leaving a usage guide that still tells people to run it.
We divide those jobs among three reviewers:
| Reviewer | Question it investigates | What it checks | |
|---|---|---|---|
| Specification | Does this deliver what we agreed to build? | The task and feature requirements, relevant architectural decisions, and tests meant to prove the required behavior. | |
| Code | Where could this implementation fail? | Concrete inputs, states or sequences of events that break the code, including whether tests detect the failures they claim to cover. | |
| Claims | Does what we say about the code still hold? | Comments, documentation, test titles and messages, including statements outside the diff that this change makes false. |
The distinction between specification and code review took some work. Both used to check whether a test's body proved its title, so they could report the same problem twice. We split that responsibility. The specification reviewer checks that required behavior has test coverage and that later test edits haven't weakened it. The code reviewer can deliberately break the implementation in its own copy of the repository and run a test to see whether the test catches the failure.
The claims reviewer also checks whether the change makes existing comments or documentation inaccurate, including in files outside the diff. If a pull request removes a command, the reviewer checks for instructions that still tell people to use it.
Those instructions need updating as part of the same change. Unrelated errors that existed before the pull request can wait for separate work.
A separate agent, the review coordinator, starts the reviewers and checks their reports. It brings their findings together and tells the implementation agent what needs fixing. The coordinator doesn't edit the code, and the reviewers investigate independently.
-1920w.webp)
We don't run every reviewer on every change. Documentation may need only claims review. A refactor intended to preserve behavior gets code and claims review, without a specification review. The coordinator chooses according to the work being done.
Make sure the reviewer receives the rules
Giving each reviewer a job description only works if it reads it. A look through our agent runs showed that specialist reviewers often skipped their instruction file. We'd written down the process, but some agents were reviewing without it.
Now the coordinator puts the instructions directly in each reviewer's prompt. It also includes examples of previously rejected findings, so the reviewer can see where earlier judgments went wrong. The files remain the source we maintain; the agent receives their contents with the assignment.
Repository rules give the reviewer something concrete to judge against. For example, Simlock requires a deadline whenever one process waits for another. It also requires destructive operations to go through device drivers. A finding can point to a broken rule and the code that breaks it. An approved specification has to follow those rules too. If it conflicts with an accepted architectural decision, we need to resolve that conflict explicitly, with the maintainer authorizing any amendment.
Some rules don't need an agent at all. The morning after that five-round pull request, I removed the comment reviewer and moved format checks into a script. A missing why: prefix can fail in a commit hook. Whether the explanation is true still needs the claims reviewer's judgment.
Check the finding before asking for a fix
When a reviewer reports a bug, the implementation agent could start fixing it immediately. But the report might be wrong, or another reviewer might have described the same defect in different words. Fixing every report would turn those mistakes into more code changes.
The coordinator first combines reports about the same defect. Each distinct finding gets an ID and a record of its evidence, decision and eventual fix. Then the coordinator reads the cited source or runs a focused reproduction to decide what should happen:
- A confirmed defect goes to the implementation agent for a fix.
- A dismissed finding keeps the evidence explaining why it was wrong, or why the behavior was intentional and broke no requirement.
- An unresolved concern with serious consequences stays open for investigation.
-1920w.webp)
The coordinator can get this wrong too. We found that while rerunning a review of an older cleanup change. The code needed to remove temporary files left by dead processes while leaving other files alone. The original review had caught a case where cleanup could touch files Simlock hadn't written. During a rerun, a reviewer noticed the risk and downgraded it to a note.
We tightened the verification rule. If a finding says the code could delete user data, expose credentials, touch files it doesn't own or mistake a running process for a dead one, the coordinator must test that case before dismissing it. Without evidence ruling it out, the concern remains blocking. A reassuring explanation isn't enough to close it.
Routine suggestions get a different treatment. We stopped collecting long lists of notes that almost nobody acted on. A naming preference or request for extra defensive code doesn't start another round. New evidence of a blocking defect can.
Check what the fix actually changed
Once a finding is confirmed, the implementation agent makes a fix. It's tempting to ask the reviewer to check that the reported problem is gone and move on. The fix itself deserves attention, especially when it changes a test.
Our earlier process divided fixes into production source and text. Tests counted as text, alongside comments and documentation, so a change to assertion could bypass another code review. That hid a fairly serious distinction: a test might turn green because the implementation works, or because the assertion that caught the bug has disappeared.
We replaced those categories with a decision based on what the edit does:
- Changed behavior or test logic goes to the code reviewer.
- A change to which requirement a test proves also goes to the specification reviewer.
- Changed explanations go to the claims reviewer.
- A purely mechanical correction can be checked by the coordinator.
-1920w.webp)
The follow-up reviewer gets the diff since the last reviewed commit, along with the previous findings, evidence and outcomes. It can check the fix and follow its effects into related code without restarting the entire review. Settled findings stay settled unless new evidence challenges them.
We also give every reviewer a fresh checkout at the commit under review. Otherwise, an experiment from an earlier check, or a file left over in the working directory, could change what the next reviewer sees.
Set a stopping point for review
One database change showed how reviews could drag on while each round appeared productive. The first round found a column that rejected fractional values the code needed to store. After the fix, the next round found another column with the same problem. We spent another cycle discovering a related mistake we could have found earlier.
Reviewers now search for other instances as soon as they confirm a defect. A wrong column type prompts a check of sibling columns; a missing guard prompts a search for similar paths. Related sites belong in the same finding so the implementer can address them in one fix. Finding a sibling in round two is recorded as a first-round miss.
I also wanted an explicit limit for deliveries like the five-round one in the opening. Our process allows:
- One full review of the change.
- A targeted second round, if fixes need review.
- One final check of the last fix, if needed.
It can finish earlier when the review is complete and no blockers or unchecked fixes remain. The pull request then goes to the merge gate, a script that checks CI and the remaining delivery requirements.
-1920w.webp)
If a blocker remains at the limit, the pull request stays in draft for me. The same applies to a fix that changes behavior or test logic and hasn't been checked. If an agent runs out of budget or never receives the inputs it needs, its assessment is incomplete, with the missing coverage stated. A report with zero findings tells me very little if the review never finished.
A regression discovered in the final check still blocks. An existing, unrelated bug can become separate work only when there's evidence it predates the change. I can then inspect what's left, authorize more work or change the plan. Reaching the round limit doesn't make the code ready to merge.
Find out what the reviewers missed
All of this tells us what to do with a reported problem. It doesn't tell us about the bugs nobody reported. Counting confirmed findings won't answer that question: the comment reviewer had all 155 confirmed, and I still removed it the next morning.
To check for misses, we added a replay harness. It takes an earlier pull request's code, base and review history, then runs a fresh review. The reviewers receive the earlier code and the requirements, while the recorded findings and subsequent fixes stay out of their inputs. Afterward, we compare their reports with the defects found during the original delivery.
-1920w.webp)
We used the cleanup and database changes from this article to test revised instructions. The follow-up replays recovered the safety finding and caught more database defects, though they still missed some. Those two snapshots give us evidence about the corrections we made. They can't tell us how often the reviewers will catch bugs in general.
We also count the cost of verification, fixes and every review call, including the final check. Usage comes from agent transcripts; missing data stays unknown. A cheaper reviewer can create more work afterward, so replacing one requires historical tests, independent reviews of live changes and tracking defects found after merge. The existing reviewer keeps the final say while we compare it with a potential replacement.
Try the handoff on one pull request
I started with stale comments and ended up reworking how agents review each other’s code. Giving reviewers the requirements and separate responsibilities was only part of it. We also had to verify their findings, check what each fix changed, and decide when to hand unresolved work back to a person. Replaying past pull requests let us investigate defects the reviewers had missed.
You can try the handoff without building the whole setup. On your next agent-written pull request, give a fresh agent the requirements and code, leaving out the implementation conversation. Ask it to identify a concrete failure case. Verify the evidence before requesting a fix, then check that the fix addresses the defect and preserves the test assertions that should catch it.
Keep a record of what the reviewer found, what it missed, and what took your time. Use that first review to decide which part of your process to change next.
Explore the code and workflow behind these examples in the Simlock repository on GitHub.
Next in the series: how an agent checks a change on a real device.






















