Will Ayers
← Back to all postsbuild notes /

Making AI Research Source-Conscious

How I built a multi-agent research system that separates sourced facts from hypotheses, preserves the work behind a decision, and makes uncertainty visible.

airesearchmulti-agenttooling

AI is useful for exploring an unfamiliar problem quickly. It is also very good at producing a confident paragraph that makes a weak claim feel settled.

I wanted a research workflow that could help with early product decisions without hiding where the information came from or what still needed verification. The result is a multi-agent research board: several focused researchers work independently, a synthesizer turns their reports into a decision memo, and the complete run stays available as files.

I call the approach source-conscious rather than source-perfect. The system cannot guarantee that every source is correct. It can require agents to show their evidence, label unsupported claims, and preserve enough context for a person to verify the result.

The technical stack

The board is a Pi extension written in TypeScript and running on Node.js. Pi provides the extension command and tool APIs. Node provides child processes, filesystem artifacts, temporary files, concurrency control, and cancellation.

The system has two entry points:

  • a /research-board command for interactive use
  • a research_board tool that an agent can call when a task explicitly needs a board-style workflow

Both entry points use the same orchestration function. This keeps the terminal command and agent tool from developing different behavior.

A run follows this pipeline:

1research brief
2  → discover agent definitions
3  → select a board mode and roles
4  → run independent research agents
5  → write one artifact per agent
6  → extract cited URLs into an index
7  → run a separate synthesizer
8  → write a decision memo and usage summary

Agents are configuration, not orchestration code

Each role is a Markdown file with a small front matter block:

1---
2name: source-scout
3description: Finds and verifies source-backed facts, URLs, and research gaps.
4tools: read,bash,grep,find,ls
5---
6

7You are a source-first research scout.
8

9Your job:
10

11- Extract facts only when supported by a source you can cite.
12- Label confidence for each finding.
13- List unresolved verification gaps.

The extension discovers global agent definitions and optional project-level overrides. Adding or changing a research perspective does not require editing the runner. A project can supply a specialist by placing another Markdown definition in its local research-board directory.

The current board grew out of product and game-economy research, so its roles include:

  • source scout
  • competitor analyst
  • game designer
  • token-economy analyst
  • legal and risk reviewer
  • holder-demand researcher
  • synthesizer

Those roles are not a universal panel. They are an example of why the agents live outside the runtime. A different project can replace them with market, customer, technical, accessibility, policy, or domain-specific researchers while keeping the orchestration unchanged.

Isolate every research run

Each researcher runs as a separate Pi process in JSON mode. The runner creates a temporary system-prompt file, starts the process without a shell, and disables sessions, extensions, skills, and prompt templates.

The process arguments are intentionally explicit:

1const args = [
2  "--mode",
3  "json",
4  "-p",
5  "--no-session",
6  "--no-extensions",
7  "--no-skills",
8  "--no-prompt-templates",
9  "--append-system-prompt",
10  temporaryPromptPath,
11];

Agent-specific model and tool settings are added after that. The temporary prompt file is created with user-only permissions and removed when the run finishes.

Isolation gives the board several useful properties:

  • researchers do not inherit unrelated conversation history
  • one agent cannot anchor another before synthesis
  • each role has an explicit tool boundary
  • process failures have a clear exit code and stderr stream
  • the parent can cancel individual processes

The runner reads Pi's newline-delimited JSON events from stdout. Assistant output, model information, stop reason, token usage, cache usage, cost, and turn count are collected from completed messages rather than scraped from terminal text.

Source-conscious behavior is part of the task contract

Every researcher receives the same behavioral requirements in addition to its role prompt:

  • cite URLs when they were provided, fetched, or otherwise known
  • label unsupported claims as hypotheses
  • state when web access is unavailable
  • separate facts, interpretation, recommendations, risks, and questions
  • return structured Markdown

The source scout is stricter. It prioritizes official material, credible third-party sources, local documents, and repository artifacts. If direct search is unavailable, it can inspect provided public URLs conservatively but must report the remaining verification gaps.

This does not make the output true by construction. It changes the failure mode. Instead of silently filling an evidence gap, the expected output contains an explicit gap that can become a follow-up task.

Bounded parallelism instead of maximum parallelism

Research roles are independent enough to run concurrently, but launching every process at once can create resource spikes and unnecessary model spend.

The board uses a small worker pool with a default concurrency of three. A shared index assigns the next role as each worker finishes:

1await Promise.all(
2  new Array(workerCount).fill(null).map(async () => {
3    while (true) {
4      const index = next++;
5      if (index >= agents.length) return;
6      results[index] = await runAgent(agents[index]);
7    }
8  }),
9);

The caller can lower or raise the limit. Cancellation is passed in as an AbortSignal. When aborted, the runner sends SIGTERM to the child process and follows with SIGKILL if it has not stopped after three seconds.

The goal is predictable execution, not the largest possible panel.

Keep the research behind the recommendation

A chat answer is easy to consume and easy to lose. Every board run creates a timestamped directory:

1.pi/research-board/runs/<timestamp>-<brief>/
2  brief.md
3  state.json
4  sources.md
5  agents/
6    source-scout.md
7    competitor-analyst.md
8    legal-risk.md
9    ...
10  synthesis.md
11  decision.md

state.json records the selected mode, agents, timestamps, and combined usage. Each agent report includes its role, description, usage, and full output. sources.md extracts URLs from the reports into one review index.

That URL file is navigation, not verification. A URL appearing in a report does not prove that the cited page supports the claim. The individual report and source still need to be checked together.

The synthesizer receives the original brief plus every agent report. Its output has a fixed decision structure:

  1. executive recommendation
  2. source-backed findings
  3. what to copy, adapt, or avoid
  4. product recommendation
  5. risks and mitigations
  6. validation plan
  7. open questions
  8. next actions

The extension writes the raw synthesis and a clean decision memo separately, then stores a pointer to the latest run.

What the system does not solve

Multi-agent research can create the appearance of consensus without actual independence. Several agents may still rely on the same weak source or the same underlying model assumptions. A synthesizer can repeat a shared error instead of detecting it.

The current source index also uses URL extraction rather than claim-level provenance. It does not yet verify that a page was fetched, preserve a snapshot, or connect a specific sentence to the exact supporting passage.

Those limitations matter. The board is useful for structured exploration and decision preparation, not as an autonomous authority.

What I would build next

The next version should make provenance more structured:

  • represent important claims with source URL, quoted evidence, retrieval time, and confidence
  • preserve fetched source snapshots or content hashes
  • detect when multiple agents rely on the same source
  • generate a contradiction matrix before synthesis
  • distinguish "not researched" from "researched but inconclusive"
  • add explicit pass/fail gates to validation-oriented runs

I also want the board to route roles based on the brief instead of relying primarily on fixed mode defaults. The right research panel should follow the decision being made.

The larger lesson is that research quality does not come from adding agents. It comes from making evidence, uncertainty, disagreement, and execution history visible enough to inspect.

Resources

These are the main technologies and references behind the system: