Will Ayers
← Back to all postssystems /

Turn Validation into Executable Agent Instructions

How to replace vague definitions of done with focused checks that AI coding agents can run, interpret, and report.

aiagentstestingvalidation

This is part four of AI Working Agreements. Start with AGENTS.md Is a Communication Protocol for the complete working-agreement structure.

"Make sure it works" is not a validation strategy.

A coding agent can interpret that instruction as running one test, compiling the project, opening the page, or simply rereading its patch. Each action may be useful, but the result is inconsistent because the expected evidence was never defined.

I use AGENTS.md to turn validation from a vague preference into executable behavior.

Validation is part of the task contract

Implementation changes the system. Validation produces evidence about that change.

Those are separate phases:

1understand → implement → validate → report

An agent should not collapse them into "the code looks correct." Reading a diff is review, not execution. A successful build proves something different from a focused behavior test. A screenshot proves something different from both.

The working agreement defines which kinds of evidence are normally expected and how to report gaps when a check cannot run.

Start with the repository's real commands

The first validation instructions should be concrete commands that are known to work:

1## Validation
2

3- Typecheck: `npm run typecheck`
4- Lint and formatting: `npm run lint`
5- Unit tests: `npm test`
6- Production build: `npm run build`

This saves the agent from rediscovering scripts and reduces failed command guesses.

Commands alone are not enough. Include preconditions and scope when they matter:

1- Run commands from the repository root.
2- Use Node 24 from `.nvmrc`.
3- Generate route types before running TypeScript directly.
4- Use `npm test -- <path>` for a focused test file.
5- The full browser suite requires the local database and is not expected for content-only changes.

Good instructions describe the path that works, not every command that exists in package.json.

Match evidence to the change

Running every check after every edit is expensive and often uninformative. Running only the fastest check can miss the behavior that changed.

I choose validation based on the affected risk surface.

Type and data-shape changes

Use typechecking, schema validation, and focused tests around serialization or parsing.

Business logic changes

Run tests for the changed behavior and adjacent edge cases. Add a test when the failure could recur and the test protects a meaningful contract.

User-interface changes

Run static checks, then inspect the actual interface at relevant viewport sizes and interaction states. Verify keyboard behavior and reduced motion when the change affects interaction.

Route and build changes

Run the production build and request the affected routes. Development mode can hide server bundling, generated type, and deployment adapter problems.

Content changes

Validate front matter, rendering, internal links, dates, and the production build. A prose-only edit does not need an unrelated test suite.

This approach keeps validation focused without treating it as optional.

Use a validation ladder

I think about checks as a ladder from cheap structural feedback to realistic behavior:

11  Formatting and lint rules
22  Typechecking and schema checks
33  Focused unit or integration tests
44  Browser or runtime behavior
55  Production build
66  Deployment preview or production verification

Not every task climbs every rung. The changed system determines where useful evidence stops.

A CSS token rename may need lint, build, and visual inspection. A payment calculation may need unit tests, integration coverage, and approval beyond technical validation. A deployment configuration change may need a preview environment even when the application test suite passes.

The ladder also helps explain what remains unknown. If browser automation is unavailable, the agent can report that levels one through three passed and level four still needs manual verification.

Write validation as executable instructions

Compare these rules:

1- Test your changes thoroughly.
1- Run the narrowest existing test that exercises changed behavior.
2- Run the project typecheck when TypeScript source changes.
3- Run the production build when routes, loaders, bundling, or configuration changes.
4- For responsive UI, verify desktop and mobile widths and check for overflow.
5- Report commands, outcomes, and anything that could not be run.

The second version gives an agent a decision procedure. It still requires judgment, but the judgment has boundaries.

Instructions can also map directories to checks:

1- Changes under `app/routes/` require route type generation and typecheck.
2- Changes under `scripts/` require running the affected script against a temporary output.
3- Changes to article front matter require confirming date order on `/blog`.
4- Changes to deployment adapters require a production build.

Path-specific instructions are especially useful in monorepos where one global command would be slow or incorrect.

Validate the user-visible claim

Agents often stop after static checks because those checks are easy to run and easy to report.

If the task says "fix the mobile menu," a passing typecheck does not validate the claim. The agent should open the menu at a mobile viewport, exercise closing behavior, and check keyboard or touch interactions that were part of the request.

If the task says "future articles should release one per day," the validation should include time-bound publication behavior:

1current article appears in the index
2future article is absent from the index
3future article route returns 404
4article becomes available after its publication timestamp

The test should follow the user-facing promise, not only the implementation mechanism.

Keep test additions proportional

AI agents can generate a large amount of test code quickly. That does not make every test valuable.

I tell agents to add focused tests for concrete risk and avoid broad regression or smoke tests without a contract to protect.

A useful test should answer at least one question:

  • What behavior failed or could plausibly fail?
  • What boundary or invariant does this test preserve?
  • Would this test fail for the broken implementation?
  • Is this the cheapest reliable level at which to test it?

Snapshotting an entire page because one label changed usually creates maintenance cost without isolating the behavior. Testing a date-normalization function across browser-sensitive inputs protects a specific contract.

Validation volume is not the goal. Decision confidence is.

Treat failures as information

A failed check does not always mean the patch is wrong. It may expose a stale fixture, an environment dependency, an unrelated existing failure, or an incorrect instruction.

The agent should diagnose before changing code to satisfy the output.

A useful failure report includes:

1Command: npm run test -- article-date
2Result: failed
3Relevant error: expected Aug 31, received Aug 30
4Scope: failure reproduces on the base commit
5Action: no unrelated test changes made
6Recommendation: fix timezone expectation separately

This separates observed evidence from the agent's interpretation and proposed action.

The working agreement should also prevent a dangerous shortcut: weakening tests, disabling rules, or adding broad ignores just to make validation green.

Report evidence precisely

"All tests pass" is only useful if the relevant tests actually ran.

I prefer concise completion reports:

1Validation
2- `npm run typecheck`: passed
3- `npm run lint`: passed
4- focused browser check at 1440px and 390px: passed
5- `npm run build`: passed
6

7Not run
8- production deployment check: deployment was not requested

This format makes the boundary of confidence visible. It also prevents a successful lint command from becoming a general claim that the feature works.

For failed or skipped validation, report the reason rather than guessing:

1- End-to-end checkout test: not run; test credentials are unavailable locally.

A known gap is better than invented certainty.

Validation is not approval

Technical evidence does not resolve product ownership, business consequence, or release authority.

A migration can pass every test and still require a rollout plan. A design can match the screenshot and still need product approval. A production build can succeed without permission to deploy it.

The validation protocol answers "what evidence do we have?" Autonomy and approval rules answer "who decides what happens next?"

Both belong in a working agreement because reliable agent behavior depends on keeping them separate.

A compact AGENTS.md section

A useful starting point:

1## Validation
2

3- Run focused checks relevant to the changed behavior.
4- Typecheck TypeScript source changes and lint files before completion.
5- Run a production build for route, loader, bundling, or configuration changes.
6- Validate user-facing behavior directly when practical.
7- Add tests for concrete contracts, not test volume.
8- Do not weaken checks to make a patch pass.
9- Report each command and result, including anything that could not be verified.

Add repository commands and path-specific rules beneath it. Remove instructions that no longer match the actual environment.

A strong validation section does not guarantee correct software. It makes the agent's evidence repeatable, inspectable, and appropriately scoped. That is a much better basis for deciding whether work is done.

Resources