The AI Agent Factory

6.5 Citation and abstention

Status
stable
6 min read
Owner
Panaversity
Approved
Panaversity ·

In everyday life. A friend says "the article says so" but cannot show you the article.

A citation points from a claim to the source that supports it: a file and row, a policy section, a web page. Abstention is the worker saying the source does not answer, instead of guessing. Both are signs of good work. When they are missing, that is a warning.

An answer that cites nothing is a warning, not proof of an error. It means you cannot check the claim without doing the work again. So mark it unsupported, look for evidence from an official source, and do not accept the claim until you find that evidence. You can call the claim fabricated, which means invented, only when the evidence shows it cannot be true. That was true of the worker's claim that Tri-County's new bank account was verified: nobody had called, and the worker had no phone. For every claim a decision rests on, ask for a citation in the Review Contract. Then run three checks on each citation.

  1. It exists. Section 3 of AP policy version 3 stops at 3.3. "Policy 3.4" fails here.
  2. It says what is claimed. Open it and read it. A real section cited for a rule it does not contain is as bad as an invented one.
  3. It is the governed version. That is the approved version in use now. A real quote from the retired version 2 still gives the wrong rule.

Citations to the web need two more checks: the page's date, and whether the page is an authority on the question.

Now look at the row for Northern Maple Paper, a supplier. The invoice is in Canadian dollars, and the policy says nothing about other currencies. The worker held it back from payment, wrote "policy v3 does not cover other currencies," and listed it for Dave, the controller, to decide. That is abstention, and it is the right answer. A reviewer who marks it "incomplete" teaches the worker to guess next time. Make abstention count as good work in your contract: "cases the policy does not cover are listed, not decided." Trust a worker that always has an answer less than one that sometimes says it cannot tell.

This is what Chapter 4 called a KSoR, a Knowledge System of Record, that cites or abstains, seen from the reviewer's side. You build one in Chapter 8.

The title reads "Check the citation. Respect the limits." The line below it reads "A decision-critical claim needs evidence you can verify." Three boxes show the three checks for every citation, each with a failing example. One, does it exist? Find the cited source or section. Fail: "Policy 3.4" does not exist. Two, does it support the claim? Read what the source actually says. Fail: a real section is cited for the wrong rule. Three, is it the governed version? Use the approved, applicable version. Fail: the quote comes from retired version 2. A gold bar reads: all three pass, the citation supports the claim. A red bar reads: no citation? Mark the claim unsupported, seek evidence, and withhold acceptance. Missing support is not proof that the claim is false. Below, under when the source is silent, abstain and escalate, one box says policy v3 does not cover CAD payments, and the brief does not authorize currency conversion. An arrow leads to the right output, in gold: "Not covered. Listed for Dave." Hold for a decision, and do not invent a rule or convert at an online rate. One footer says that correct abstention is good work when the source truly does not answer. Another says that for web sources, you also check the date and the source's authority.

Figure 6.5. Three checks on every citation, and what to do when the source has no answer.

Check yourself

Question 1 / 8 · blueprint

0 answered

What does an answer that cites nothing tell you?

On this page