Building a Multi-Agent Code Review Board

I wanted a better way to review large or risky changes than asking one AI agent to "review this repository."

That prompt can produce useful feedback, but it has a structural problem: one reviewer has to think like an architect, security engineer, test engineer, reliability engineer, and domain expert at the same time. The result is often broad but shallow. Findings can also arrive without enough evidence to distinguish a real defect from a plausible concern.

I built a code review board to make that process more deliberate. It runs several independent reviewers with narrow responsibilities, then gives a final agent the job of verifying, deduplicating, and prioritizing their findings.

The implementation lives inside my coding environment as an extension. The extension is the entry point, but most of the work is in the review runtime around it.

The technical stack

The board is a Pi extension written in TypeScript and running on Node.js. Pi supplies the extension lifecycle, model runtime, agent sessions, tool registration, and terminal UI. Node handles process execution, filesystem persistence, locking, timeouts, and cancellation.

The main Pi APIs I use are:

• createAgentSession to create an isolated session for each reviewer
• ModelRuntime to resolve authenticated models at runtime
• DefaultResourceLoader to give each reviewer a controlled system prompt
• SessionManager.inMemory to keep reviewer conversations isolated
• pi.exec to run repository validation commands with cancellation and time limits
• extension commands and UI status methods to start runs and report progress

The reviewers currently use configurable OpenAI Codex models. Model choice is separate from reviewer behavior, so a role can move to a different model without changing the orchestration.

At a high level, one run looks like this:

1command
2  → inspect repository signals
3  → choose review profile
4  → capture Git and dependency context
5  → run validation commands
6  → launch independent read-only reviewers
7  → persist reviewer artifacts
8  → run evidence-aware synthesis
9  → write the final Markdown report

Start with independent perspectives

The board has a core set of reviewers:

• code correctness and data integrity
• software architecture
• application security and supply chain
• testing and quality
• reliability and performance

It can add specialists for frontend accessibility, databases and migrations, data or machine learning, compliance, and project-specific domains.

Each reviewer gets its own isolated session, focused instructions, and read-only tools. Reviewers inspect the same repository independently instead of seeing and reinforcing one another's conclusions.

That independence matters. Agreement is more useful when it comes from separate inspection. Disagreement is useful too because it tells the synthesizer where evidence needs another look.

Route the review instead of running everything

More agents do not automatically produce a better review. They can add cost, repeat findings, and manufacture concerns outside their expertise.

The board supports explicit profiles for quick, standard, web, data, regulated, and full reviews. It can also select a profile automatically from the requested scope and repository signals.

A React application can bring in the frontend and accessibility reviewer. A project with an ORM or migration files can bring in the database reviewer. Financial or privacy-sensitive language can add a compliance perspective.

The goal is not maximum coverage on every run. It is the smallest review group that covers the actual risk surface.

Give every reviewer the same evidence

Before launching reviewers, the system creates a bounded review context containing:

• the current commit and working-tree status
• changed files and a diff summary
• a bounded diff snapshot
• relevant dependency and project manifests
• results from detected validation commands

This fast-start artifact keeps each reviewer from spending most of its time rediscovering repository shape. It is only an index, not proof. Reviewers still have to inspect source directly before reporting a finding.

That distinction became one of the board's central rules: context helps an agent navigate, but source is the evidence.

Each reviewer session is intentionally constrained:

1const { session } = await createAgentSession({
2  cwd,
3  model,
4  resourceLoader,
5  tools: ["read", "grep", "find", "ls"],
6  sessionManager: SessionManager.inMemory(cwd),
7  settingsManager: SettingsManager.inMemory({
8    compaction: { enabled: true },
9  }),
10});

Reviewers cannot edit files or execute shell commands. Extensions, skills, prompt templates, and themes are also disabled inside these child sessions. That removes unrelated context and makes the role prompt the primary source of behavior.

Validation happens outside those reviewer sessions. The board detects conventional project commands such as typechecking, linting, and tests, then runs them through pi.exec. Full output goes to bounded log files while reviewers receive concise excerpts and exit status. This keeps a large test log from consuming the review context.

Durable run artifacts

Every run gets its own directory instead of existing only in a chat transcript:

1.pi/code-review-board/runs/<timestamp>-<scope>/
2  manifest.json
3  state.json
4  review-context.md
5  validation.md
6  validation-logs/
7  agents/
8  transcript.md
9  synthesis.md

The manifest records the scope, selected profile, reviewers, routing reasons, and configuration. Each reviewer writes its own report before synthesis begins. This makes partial runs inspectable and gives the synthesizer stable files to read instead of passing every result through one large prompt.

A small throttled state writer updates the dashboard without writing on every streamed token. A lock file created with exclusive filesystem semantics prevents concurrent board runs on the same machine. Cancellation uses AbortController, and active child sessions are explicitly aborted when the total timeout is reached.

Make failure visible

Multi-agent systems have more ways to fail than a single prompt. An agent can time out, exhaust its budget, return incomplete output, or disappear during synthesis.

I did not want those failures to look like a successful review.

Every run tracks reviewer status, cost, output limits, and timeouts. The board caps concurrency and total execution. It writes intermediate state and individual reviewer artifacts to disk. A machine-level lock prevents multiple expensive boards from running over each other.

If synthesis cannot complete, the board produces an explicit incomplete report instead of quietly treating partial coverage as approval. Missing evidence should reduce confidence, not disappear from the final output.

Synthesis is a verification step

The final agent is not asked to concatenate summaries. It has a narrower job:

1. Read the validation results and independent reports.
2. Group duplicate findings by root cause.
3. Inspect source when evidence conflicts or remains ambiguous.
4. Keep severity attached to demonstrated impact.
5. Report which reviewers completed, failed, or were skipped.
6. Produce a concrete fix plan.

The output is a Markdown report with an executive verdict, scope, validation results, severity-ranked findings, reviewed strengths, limitations, and residual risk.

The "reviewed strengths" section is intentional. A useful review should say which important areas were inspected without producing an actionable finding. Otherwise readers cannot tell the difference between "checked and acceptable" and "never looked at."

What changed after using it

The first version was essentially a panel of reviewers. The system became useful when I added operational constraints around the agents:

• role-specific routing instead of always running the full panel
• safe, read-only reviewer sessions
• automatic validation before subjective review
• fast-start context to reduce repeated discovery
• bounded concurrency, output, time, and spend
• durable artifacts for every stage
• explicit incomplete states

Those features are less exciting than adding another model, but they determine whether the result can support an engineering decision.

What I would build next

I want to improve how the board learns from previous reviews without teaching it to repeat old findings. I am also interested in measuring which reviewer combinations find verified issues for different kinds of repositories.

The larger lesson is straightforward: using several agents is easy. Designing the boundaries, evidence flow, failure states, and final decision process is the actual work.

Resources

These are the main projects and documentation behind the system:

• Pi agent harness — the coding-agent environment and extension runtime
• Pi extension documentation — lifecycle events, custom commands, tools, sessions, and UI APIs
• Pi source code — implementation of the coding agent, model runtime, and agent sessions
• TypeScript — the extension and orchestration language
• Node.js — filesystem, process, cancellation, and runtime APIs
• Git — repository state and bounded diff context
• OpenAI Codex — the model family currently assigned across reviewer roles