The AI Agent Factory

6.1 Agree the checks before you delegate

Status
stable
9 min read
Owner
Panaversity
Approved
Panaversity ·

In everyday life. Before a painter starts on your kitchen, you agree on the job: two coats, no drips on the counters, the old color fully covered, and stop if the wall is wet.

A Review Contract is the acceptance criteria you set before delegation: the tests the work must pass before you accept it. It answers four questions. The last column answers them for Brightline's payment run on Friday, October 23: the bills that Dave, the controller, approves for payment that day.

QuestionWhat to writeFor Friday's run
What must be checked?The facts, figures and decisions that make the work usableEvery open invoice has one action that follows AP policy version 3, Brightline's current rules for paying bills. Every amount, date and approval matches the source. The totals Dave signs
What evidence comes back?What the worker returns, so you can check without doing the work againA reason and a policy section on every row of the proposal file. The approval ID from the approvals log for every approval. The task record, the worker's log of what it did. Who verified anything the output says was verified
What counts as success?Tests the output must passEach of the 15 invoices appears exactly once. The totals match totals worked out again from the source files, to the cent. The proposal file, the memo and the note agree. Every claim traces to a source
What stops the worker?Cases where it must stop and askA request to change payment details. Inputs that disagree, when the brief does not say which one wins. A case the policy does not cover

Write it first, as its own document. Then share all four parts with the worker. Two of them must also go into the brief, because the worker acts on them: the evidence to return and the stop conditions (Chapter 5). The brief can include the success criteria too, so the worker checks its own work before it returns it. A worker that checks itself has done useful work, but that is not an independent review. The independent check and the final decision stay with you.

Why first? Once you have seen a result, you compare it only with itself. A well-organized memo sets its own standard, and its gaps become hard to see. Psychologists call this anchoring: the first confident answer becomes the starting point for your later judgment.1 This book's first edition has a Prediction Lock, which applies the same idea to decisions: write down what you think before AI answers. A Review Contract is the Prediction Lock for delegated work. It also shows you a bad brief early. If you cannot say what success looks like, the worker cannot either.

Before delegation or before inspection. Writing first helps most before delegation, when the contract can still shape the work. If someone hands you work that is already done, write the contract before you open it. It can no longer shape the work, but it still protects you from anchoring.

How much review? Match the depth of the review to what is at risk. Ask three questions. Can the result be undone? Will it leave the company, or change a record of truth, such as the vendor record, where each supplier's details are kept? Will someone act on it without checking it again?

If the work...Review depthExample at Brightline
Moves money, goes outside the company, or changes a record of truthFull contract, every check, a named human signsThe payment run, a vendor email, a change to the vendor record
Feeds a decision someone else makesFull contract on the numbers and claims the decision rests on, and a sample of the restThe memo Dave uses to approve the run
Repeats often, with little at riskA standing contract that you reuse, with a person checking a sample each weekGiving each week's office-supply invoices their account codes
Is a draft you will rewrite yourselfA light check as you rewrite itMaria's first draft of a vendor letter

When the answer is unclear, choose the deeper level. Chapter 10 sets the cases where a human must sign, whatever the review finds. There is one more question: what must never happen automatically? In this book that question belongs to the Authority Envelope, the outer limit of what the worker may ever do (Chapter 7). It limits the worker, not the review.

The title reads "The Review Contract comes first," and the line below it reads "Agree the standard before you delegate." A box on the left is headed Review Contract: write first, share all four parts. It lists the four parts. One, what must be checked, shared with the worker. Two, what evidence comes back, must be included in the brief, in gold. Three, what counts as success, shared with the worker. Four, what stops the worker, must be included in the brief, in gold. An arrow labeled share the whole contract leads to the brief, from Chapter 5. Next is the AI Worker, which executes the task. Then an arrow leads down to the output and task record, what comes back. That goes to the human reviewer: verify against sources, decide, sign when required. A dashed line from the contract to the reviewer reads: independent verification and the verdict stay with you. A footer reads: share the criteria, independently verify the result. Below it: inherited work? Write the contract before opening the output.

Figure 6.1. The Review Contract comes first.

Check yourself

Question 1 / 8 · current

0 answered

Which four questions does a Review Contract answer?

  1. How to Think in the AI Era, Panaversity, first edition.↑

On this page