← Back to all postssystems /

AI Makes Production Cheap. Judgment Becomes the Bottleneck.

When AI makes drafts and options easier to produce, the useful constraint becomes our capacity to evaluate them and take responsibility for the result.

aijudgmentdecision-making

I can ask for another draft more easily than I can decide whether the first one is worth using.

That difference changes how I want to work with AI. If producing material becomes faster but evaluating it remains difficult, more production can leave me with more unfinished work. The documents exist. The decisions do not.

The title needs a boundary: AI does not make all production cheap. Reliable research, sensitive data handling, specialist work, and physical delivery can remain expensive. Even a short document may take substantial effort to get right. But in tasks where generating a plausible draft costs little, judgment can become the limiting resource.

This second article in Human Systems for AI Work is about that shift. The first argued for a repeatable system around AI rather than dependence on a perfect prompt. Here the question is how to use the time saved on production without creating a review burden that cancels the benefit.

Output is not the same as progress

Imagine a team choosing a customer research approach. AI produces twelve possible studies, a comparison table, sample questions, and a proposed schedule. The response is impressive in scope.

The team still has not decided what it needs to learn, which customers it can reach, or what decision the research will change. Now it has several pages of proposals to evaluate before answering those questions.

The generation was fast. The task did not necessarily move forward.

I want to distinguish between producing material and reducing the uncertainty that prevents action. A short recommendation that identifies one unresolved assumption may do more for the work than a complete plan built around an assumption nobody has accepted.

This distinction changes the request. Instead of asking for every possible approach, ask which choices materially affect the decision and what information would help choose between them.

The useful output may be smaller because the work is better defined.

Judgment has several parts

“Use human judgment” can sound like an answer while leaving the difficult work unspecified.

In practice, a reviewer may need to determine whether the facts are supported, whether the proposed action fits the situation, whether the tradeoffs are acceptable, and whether the person making the decision has authority to make it. These are different questions.

A finance specialist can verify a cost estimate without knowing whether a customer will accept the proposal. A customer lead can judge the relationship without being qualified to approve a legal commitment. Someone may recognize elegant writing but lack the subject knowledge to detect a false claim.

I do not want a single approval to hide these distinctions. For consequential work, identify which judgment is required and who can actually provide it.

That does not mean assembling a committee for every document. It may mean one knowledgeable person reviewing the decisive claim rather than three people editing the introduction.

Establish criteria before reading the recommendation

A polished answer provides its own structure for evaluation. It names the alternatives, chooses the comparison categories, and gives one option a persuasive explanation. If I accept that structure without examining it, I have delegated part of the decision before I realize it.

I prefer to establish a few criteria first.

For the research example, the study needs to answer a specific question, reach relevant participants, and finish before a pending decision. A method that produces rich information after the decision date may be unsuitable even if the report describes it as rigorous.

Criteria should include disqualifying conditions, not just preferences. If the team cannot lawfully access the required data, an otherwise attractive method is not an available option. If nobody can act on the result, more detail will not make the work useful.

The criteria can change when new information appears. What matters is noticing the change and deciding whether it is justified, rather than quietly adopting whatever standard makes the generated recommendation look strongest.

Ask for fewer, materially different options

Ten options are not necessarily ten choices. They may be small variations on the same idea, distinguished mostly by names and phrasing.

When I need alternatives, I want them to expose real tradeoffs. One research approach might be faster but cover fewer customers. Another might offer broader coverage but provide less depth. A third might require information we do not currently have.

That is enough to start a useful decision. Additional options should earn their review cost by revealing a different constraint or a plausible better outcome.

This is also a reason to avoid requesting a complete deliverable for every possibility. Compare short outlines first. Develop the chosen direction after the consequential differences are understood.

AI makes it easy to elaborate an unchosen option. That does not make the elaboration free for the people who have to read it, discuss it, and remember why it was rejected.

Spend review effort where an error would matter

Not every sentence deserves the same depth of review.

In a recommendation memo, a single unsupported assumption about customer demand may matter more than several pages of background. In a budget proposal, one mistaken recurring cost can matter more than the clarity of the executive summary.

I want the review to start with the claims that could change the decision. Verify those against appropriate sources or responsible people. Then examine whether the recommendation actually follows from them. Only after that does polishing the presentation become useful.

Sampling can be reasonable for low-consequence, repetitive material. It is not a substitute for checking a unique claim on which a major decision depends. Nor does finding no error in one section establish that an unrelated section is reliable.

A practical review question is: if this statement were wrong, what would we do differently? The answer helps allocate attention. It is more useful than treating every paragraph as equally risky because AI helped write it.

AI can assist review without settling it

AI can help identify contradictions, compare a proposal against explicit criteria, and point out where evidence appears to be missing. Those are useful ways to reduce the work a reviewer must organize.

I still would not treat an AI review as independent confirmation simply because it arrived in a separate response. Two answers can share the same mistaken premise. A critique can sound demanding while missing the only issue that matters.

The more important question is what new evidence the review introduces. Did it check the original source? Did it surface a relevant constraint? Did it identify a claim that a qualified person must verify?

Asking for criticism is worthwhile when it produces inspectable questions. It is less useful when it produces another confident verdict that now also needs review.

The reviewer remains responsible for deciding whether the checks are appropriate. Automating part of that work should make the basis for the decision easier to see, not make ownership less clear.

Limit the amount waiting for judgment

If a team can responsibly evaluate two proposals this week, generating twenty proposals does not create twenty opportunities ready to use. It creates a queue.

A small queue can be useful. People need work available when they are ready. A large queue becomes expensive when assumptions change before review or when someone has to keep sorting material that was never important enough to evaluate.

I would rather limit active work to what the team can finish evaluating. When review fills up, the next useful action may be clarifying a criterion, obtaining a missing source, or withdrawing a low-priority request. It is not automatically generating more.

This requires permission to stop. If every request must produce a polished artifact, teams have little room to decide that the question is premature or the answer would not change anything.

A deliberate “not worth developing” is a completed decision. It should not be treated as a failure to exploit the tool.

Protect the practice that builds judgment

There is a tension in delegating the first draft. Reading a finished answer can be efficient, but doing the initial analysis is sometimes how a person learns what to notice.

I do not assume the same arrangement is right for every task or level of experience. When learning a subject, I would sometimes make my own assessment before viewing an AI recommendation. Then I can compare the reasoning, identify what I missed, and investigate the disagreement.

For routine work I already understand, that extra step may add little. For a new area, it can reveal whether I am evaluating the answer or merely recognizing familiar language.

Teams also need experienced reviewers to explain consequential corrections. “This is wrong” fixes one artifact. Explaining which assumption failed gives the next reviewer something they can apply independently.

The aim is not to preserve manual effort for its own sake. It is to avoid eliminating the learning activity while assuming the resulting expertise will remain available.

Measure the finished work, not just the fast step

A claim that AI saved an hour on drafting is incomplete if review required another two hours. It is equally incomplete to count only review time and ignore a useful draft that would otherwise have taken a day.

For a recurring task, examine the whole sequence: preparation, production, review, correction, and use. Notice whether the result was accepted, whether important errors escaped, and whether the eventual decision was useful. The measures can be simple; they should cover more than the fastest visible step.

Judgment will not always be the bottleneck. A team may already have clear standards and spare review capacity. The limiting factor may be access to customers, approval from another organization, or the ability to carry out a decision. In that case, constrain and improve the actual limiting step rather than assuming every AI workflow has the same problem.

The argument is conditional: when production becomes easier than responsible evaluation, optimize for evaluation and completion.

Give the next draft a review budget

Before requesting the next substantial piece of AI work, name the decision it supports, the person who can evaluate it, and the few claims that need the most scrutiny.

Then request only enough material to make that decision well. Expand it when the review reveals a reason, not because generating more is easy.

I want the benefit of AI to show up as useful work completed with appropriate confidence. A larger pile of plausible answers is not the same result.