Building a Software Factory: AI Reviewers That Don’t Share the Writer’s Blind Spots

Part 3 of The React Native Software Factory, a series about tuning the setup around coding agents until you can leave it alone for a night.

In October, I was trying to stop comments from going stale in Simlock, a tool we build at Callstack that lets agents lease simulators and physical devices for testing.

The coding agents I use to develop it were changing the code while leaving behind comments that described how it used to work. I decided to give a separate reviewer agent the job of catching that.

I wanted to check that comments still explained the code accurately, including why it behaved unexpectedly.. We added a rule, including a required why: prefix, for the reviewer to enforce. On the next large pull request, it raised 155 findings marked as blockers, each of which had to be resolved before the change could proceed.

Another agent verified them, the implementation agent made the fixes, and the pull request went through five review rounds.

I'd added this reviewer to improve the code, and now I had another queue of work to manage. Even a correct finding needs someone to verify it, make a fix and check that fix. Reviewers can also miss defects, so the number of findings alone couldn't tell me how well the review was working.

That experience led me to rethink the whole review process: what a fresh agent should see, which questions it should investigate, and how to get from a reported problem to a finished change. How could I get a reviewer to challenge the writer's assumptions without keeping the two of them busy forever?

In this article, I’ll show how we give independent reviewers distinct jobs, verify their findings, and recheck fixes within a limited number of rounds. I’ll also explain how we replay past pull requests to investigate what the reviewers missed.

Start with an AI reviewer who hasn't written the code

In Simlock, an implementation agent starts from an approved GitHub issue. For a feature, the issue describes one task and links to the larger feature plan. The agent writes code and tests, then opens a draft pull request. The previous article covered how we prepare that task; this is where we check whether the code delivers it.

I want another agent to investigate the change without having followed the conversation that produced it. The writer has already settled on an approach and explained why it should work (we talked about it in the previous part of this series). Passing that explanation along gives the reviewer a ready-made interpretation before it has read the code.

So the reviewer receives the code, the requirements and the repository's rules, with the code fixed at a specific commit. It doesn't receive the implementation conversation, pull request description or commit messages. It has to work out how the implementation behaves from the source.

In an earlier version of this setup, we also withheld the specification from the code reviewer. It kept objecting to choices we'd deliberately made during planning, so we added the specification to its inputs. The reviewer needed to know what we’d agreed to build while investigating the implementation independently.

Give the reviewers different jobs

Checking whether we built the requested feature is different from finding a way to break its implementation. Neither check necessarily catches outdated documentation: we might remove a command while leaving a usage guide that still tells people to run it.

We divide those jobs among three reviewers:

Reviewer Question it investigates What it checks
Specification Does this deliver what we agreed to build? The task and feature requirements, relevant architectural decisions, and tests meant to prove the required behavior.
Code Where could this implementation fail? Concrete inputs, states or sequences of events that break the code, including whether tests detect the failures they claim to cover.
Claims Does what we say about the code still hold? Comments, documentation, test titles and messages, including statements outside the diff that this change makes false.

The distinction between specification and code review took some work. Both used to check whether a test's body proved its title, so they could report the same problem twice. We split that responsibility. The specification reviewer checks that required behavior has test coverage and that later test edits haven't weakened it. The code reviewer can deliberately break the implementation in its own copy of the repository and run a test to see whether the test catches the failure.

The claims reviewer also checks whether the change makes existing comments or documentation inaccurate, including in files outside the diff. If a pull request removes a command, the reviewer checks for instructions that still tell people to use it.

Those instructions need updating as part of the same change. Unrelated errors that existed before the pull request can wait for separate work.

A separate agent, the review coordinator, starts the reviewers and checks their reports. It brings their findings together and tells the implementation agent what needs fixing. The coordinator doesn't edit the code, and the reviewers investigate independently.

We don't run every reviewer on every change. Documentation may need only claims review. A refactor intended to preserve behavior gets code and claims review, without a specification review. The coordinator chooses according to the work being done.

Make sure the reviewer receives the rules

Giving each reviewer a job description only works if it reads it. A look through our agent runs showed that specialist reviewers often skipped their instruction file. We'd written down the process, but some agents were reviewing without it.

Now the coordinator puts the instructions directly in each reviewer's prompt. It also includes examples of previously rejected findings, so the reviewer can see where earlier judgments went wrong. The files remain the source we maintain; the agent receives their contents with the assignment.

Repository rules give the reviewer something concrete to judge against. For example, Simlock requires a deadline whenever one process waits for another. It also requires destructive operations to go through device drivers. A finding can point to a broken rule and the code that breaks it. An approved specification has to follow those rules too. If it conflicts with an accepted architectural decision, we need to resolve that conflict explicitly, with the maintainer authorizing any amendment.

Some rules don't need an agent at all. The morning after that five-round pull request, I removed the comment reviewer and moved format checks into a script. A missing why: prefix can fail in a commit hook. Whether the explanation is true still needs the claims reviewer's judgment.

Check the finding before asking for a fix

When a reviewer reports a bug, the implementation agent could start fixing it immediately. But the report might be wrong, or another reviewer might have described the same defect in different words. Fixing every report would turn those mistakes into more code changes.

The coordinator first combines reports about the same defect. Each distinct finding gets an ID and a record of its evidence, decision and eventual fix. Then the coordinator reads the cited source or runs a focused reproduction to decide what should happen:

  • A confirmed defect goes to the implementation agent for a fix.
  • A dismissed finding keeps the evidence explaining why it was wrong, or why the behavior was intentional and broke no requirement.
  • An unresolved concern with serious consequences stays open for investigation.

The coordinator can get this wrong too. We found that while rerunning a review of an older cleanup change. The code needed to remove temporary files left by dead processes while leaving other files alone. The original review had caught a case where cleanup could touch files Simlock hadn't written. During a rerun, a reviewer noticed the risk and downgraded it to a note.

We tightened the verification rule. If a finding says the code could delete user data, expose credentials, touch files it doesn't own or mistake a running process for a dead one, the coordinator must test that case before dismissing it. Without evidence ruling it out, the concern remains blocking. A reassuring explanation isn't enough to close it.

Routine suggestions get a different treatment. We stopped collecting long lists of notes that almost nobody acted on. A naming preference or request for extra defensive code doesn't start another round. New evidence of a blocking defect can.

Check what the fix actually changed

Once a finding is confirmed, the implementation agent makes a fix. It's tempting to ask the reviewer to check that the reported problem is gone and move on. The fix itself deserves attention, especially when it changes a test.

Our earlier process divided fixes into production source and text. Tests counted as text, alongside comments and documentation, so a change to assertion could bypass another code review. That hid a fairly serious distinction: a test might turn green because the implementation works, or because the assertion that caught the bug has disappeared.

We replaced those categories with a decision based on what the edit does:

  • Changed behavior or test logic goes to the code reviewer.
  • A change to which requirement a test proves also goes to the specification reviewer.
  • Changed explanations go to the claims reviewer.
  • A purely mechanical correction can be checked by the coordinator.

The follow-up reviewer gets the diff since the last reviewed commit, along with the previous findings, evidence and outcomes. It can check the fix and follow its effects into related code without restarting the entire review. Settled findings stay settled unless new evidence challenges them.

We also give every reviewer a fresh checkout at the commit under review. Otherwise, an experiment from an earlier check, or a file left over in the working directory, could change what the next reviewer sees.

Set a stopping point for review

One database change showed how reviews could drag on while each round appeared productive. The first round found a column that rejected fractional values the code needed to store. After the fix, the next round found another column with the same problem. We spent another cycle discovering a related mistake we could have found earlier.

Reviewers now search for other instances as soon as they confirm a defect. A wrong column type prompts a check of sibling columns; a missing guard prompts a search for similar paths. Related sites belong in the same finding so the implementer can address them in one fix. Finding a sibling in round two is recorded as a first-round miss.

I also wanted an explicit limit for deliveries like the five-round one in the opening. Our process allows:

  1. One full review of the change.
  2. A targeted second round, if fixes need review.
  3. One final check of the last fix, if needed.

It can finish earlier when the review is complete and no blockers or unchecked fixes remain. The pull request then goes to the merge gate, a script that checks CI and the remaining delivery requirements.

If a blocker remains at the limit, the pull request stays in draft for me. The same applies to a fix that changes behavior or test logic and hasn't been checked. If an agent runs out of budget or never receives the inputs it needs, its assessment is incomplete, with the missing coverage stated. A report with zero findings tells me very little if the review never finished.

A regression discovered in the final check still blocks. An existing, unrelated bug can become separate work only when there's evidence it predates the change. I can then inspect what's left, authorize more work or change the plan. Reaching the round limit doesn't make the code ready to merge.

Find out what the reviewers missed

All of this tells us what to do with a reported problem. It doesn't tell us about the bugs nobody reported. Counting confirmed findings won't answer that question: the comment reviewer had all 155 confirmed, and I still removed it the next morning.

To check for misses, we added a replay harness. It takes an earlier pull request's code, base and review history, then runs a fresh review. The reviewers receive the earlier code and the requirements, while the recorded findings and subsequent fixes stay out of their inputs. Afterward, we compare their reports with the defects found during the original delivery.

We used the cleanup and database changes from this article to test revised instructions. The follow-up replays recovered the safety finding and caught more database defects, though they still missed some. Those two snapshots give us evidence about the corrections we made. They can't tell us how often the reviewers will catch bugs in general.

We also count the cost of verification, fixes and every review call, including the final check. Usage comes from agent transcripts; missing data stays unknown. A cheaper reviewer can create more work afterward, so replacing one requires historical tests, independent reviews of live changes and tracking defects found after merge. The existing reviewer keeps the final say while we compare it with a potential replacement.

Try the handoff on one pull request

I started with stale comments and ended up reworking how agents review each other’s code. Giving reviewers the requirements and separate responsibilities was only part of it. We also had to verify their findings, check what each fix changed, and decide when to hand unresolved work back to a person. Replaying past pull requests let us investigate defects the reviewers had missed.

You can try the handoff without building the whole setup. On your next agent-written pull request, give a fresh agent the requirements and code, leaving out the implementation conversation. Ask it to identify a concrete failure case. Verify the evidence before requesting a fix, then check that the fix addresses the defect and preserves the test assertions that should catch it.

Keep a record of what the reviewer found, what it missed, and what took your time. Use that first review to decide which part of your process to change next.

Explore the code and workflow behind these examples in the Simlock repository on GitHub.

Next in the series: how an agent checks a change on a real device.

Insights

Learn more about AI

Nov 12Live Stream

Mobile Development, Plugins & AI-Powered Workflows

Join Kewin Wereszczyński and Hanno Gödecke on November 12 for live demos and a discussion of the Mobile Dev plugin and AI-powered mobile workflows.

Building the React Native Software Factory — cover

Nov 11Talk

Building the React Native Software Factory

Join us at React Foundation Meetup London on November 11 to explore how AI agents build and debug React Native apps, from one terminal to the cloud.

Build, Launch, and Maintain: The Business Case for a Software Factory

Nov 10Article

Build, Launch, and Maintain: The Business Case for a Software Factory

Explore an anonymized software factory business case, its delivery workflow, and the staffing assumptions behind a projected 65% reduction in total costs.

Harness Engineering: Context, Skills, and Reliable AI Workflows

Nov 10Podcast

From Prompts to Harnesses: Building Reliable AI Workflows

Amazon’s Anisha Malde joins Łukasz Chludziński to discuss harness engineering, agent skills, and how AI is changing cross-platform development.

Building a Software Factory: Write Specs an Agent Can Build From — cover

Nov 6Article

Building a Software Factory: Write Specs an Agent Can Build From

Learn how to research requirements, resolve open decisions, and write testable specs that AI coding agents can implement without the planning conversation.

 Large-Scale Migrations With Coding Agents: Planning, Verification, and Rollback — cover

Nov 4Article

Large-Scale Migrations With Coding Agents: Planning, Verification, and Rollback

Plan codebase migrations with coding agents using reviewable requirements, verification checks, staged rollouts, and defined rollback criteria.

Conversational Banking: From Customer Questions to Authorized Actions — cover

Nov 2Article

Conversational Banking: From Customer Questions to Authorized Actions

Explore conversational banking architecture through a card-dispute example, covering tool access, authorization, execution, observability, and human handoff.

Building a Software Factory: The Night My Agents Waited for Me — cover

Oct 28Article

Building a Software Factory: The Night My Agents Waited for Me

Learn how permissions, review limits, and missing product decisions stalled two overnight coding agent runs, and how their maintainer revised the workflow.

Your AI Coding Agent Still Needs an Architecture — cover

Oct 22Article

Your AI Coding Agent Still Needs an Architecture

Use explicit contracts, behavior tests, and dependency checks to guide AI coding agents and review their changes through a TypeScript favorites example.

cover

Oct 15Live Stream

Who Left the Agents Running? React Native Software Factories

Join Kewin Wereszczyński and Szymon Chmal to explore what can go wrong in a React Native software factory overnight and the controls that keep agents on track.

cover

Oct 14Podcast

When Coding Gets Faster, What Breaks Next?

Matt Hargett explains how AI moves bottlenecks across testing, deployment, infrastructure, support, and the rest of software delivery.

cover

Oct 12Podcast

AI Coding Agents and Hermes With Tzvetan Mikov

Tzvetan Mikov discusses AI coding agents, context switching, low-level engine optimization, reviewing specs and tests, and his plans for Hermes X.

cover

Mar 19Tutorial

The Hard Way vs. The React Native AI SDK Way: Stop Writing Custom Modules for Every Model

Learn how React Native AI uses a unified provider model to switch between cloud and on-device LLMs without rewriting application logic, enabling reliable AI experiences online and offline.

cover

Mar 12Tutorial

Build Smarter Apps: Tool Calling & AI Orchestration Explained

Learn how on-device models in React Native can call tools, interact with external APIs, and return structured outputs you can use directly in application logic.

cover

Mar 5Tutorial

MLC LLM + React Native: On-Device AI Without the Pain

Learn how to run third-party on-device LLMs in React Native using MLC LLM. Choose your models, run them efficiently across platforms, and keep a unified JavaScript API with React Native AI.

cover

Feb 24Tutorial

Intelligent Fallbacks with Apple Intelligence in React Native

Learn how to use Apple’s built-in Foundation Models in React Native apps, reduce memory usage, and enable fast, fully on-device AI with React Native AI Apple.

cover

Feb 10Tutorial

What Is the React Native AI SDK? A Complete Intro & Quickstart

Learn why on-device AI matters for React Native apps, how local LLMs behave offline, and what problems React Native AI is designed to solve from day one.

cover

Jan 29Tutorial

Implementing an Android TurboModule with Kotlin

Learn how to implement a React Native TurboModule on Android, from the TypeScript spec and Codegen to the Kotlin implementation and package registration.

cover

Jan 19Tutorial

Working with Different Threads in Swift TurboModules

Learn how JavaScript, native module, background, and main threads interact in React Native TurboModules, and how to use GCD queues to run async work safely in Swift.

cover

Jan 9Tutorial

How to Add Type-Safe Constants to Swift TurboModules

Learn how to expose typed, immutable constants from a Swift TurboModule using Codegen, and get full end-to-end type safety between native and JavaScript.

cover

Dec 30Tutorial

Adding Event Emitters to Your TurboModule in Swift

Learn how to emit native events from a Swift TurboModule and subscribe to them in JavaScript, including proper memory management and Codegen integration.

Writing a Swift TurboModule in React Native

Dec 22Tutorial

Writing Your First TurboModule in Swift

Learn how to build a fully working TurboModule in Swift, integrate it with Codegen, and bridge it through Objective‑C inside an Expo app.