Tutorial: a complete review, offline

This needs part 1 of the install and nothing else. No API key, no account, and after the install no network.

The tutorial runs a whole review once, in your agent, on a public paper that ships with refereekit. The paper is the author’s own arXiv:2106.00185, included under its CC BY-NC-SA 4.0 license. The questions and answers are fixed text, and no model runs. So nothing here judges the paper, or shows how well a model reviews. What is real is everything refereekit does around the model. It reads the PDF, checks each piece of evidence against it, and checks the drafts.

The tutorial’s text is public, but it appears the way a real review’s does: in your private view, never in the chat. So it is also practice in where to look.

1. Run it

Ask your agent for the tutorial: /refx-tutorial in pi or Claude Code, $refx-tutorial in Codex, or in plain words in a desktop app. In pi the transcript opens in an overlay. In the other agents the review window opens at its Tutorial part. Each run is a fresh session, so run it as often as you like.

The first question’s answer reads:

statement 1 (s1):
  > The paper builds its basic object from a set of nodes.
  e1 quotation: PASS: the quoted words occur on the named page; inputs {"page": 1, "text": "a finite set of nodes"}
statement 2 (s2):
  > The model is set up after the introduction.
  e2 page: FLAG: page 2 exists; no quotation to verify: 0 words, need 4; a pointer to the page only, its wording is unchecked; inputs {"page": 2}
statement 3 (s3):
  > This practice quotation is not in the paper.
  e3 quotation: FAIL: not found on page 3; inputs {"page": 3, "text": "words that are not on that page"}

Each answer comes as statements. The lines marked > are the model’s own words, and they are not checked. Under each statement is the evidence it links, such as a quotation with its page, and what the check found.

2. The three verdicts

The first answer is written to show all three:

PASS. The words “a finite set of nodes” are on page 1. A PASS means the quoted words occur on the named page, and nothing more. It does not say the statement is true.
FLAG. The answer points at page 2 but quotes nothing. Page 2 exists, but its wording is unchecked. Read it yourself.
FAIL. The quoted words are not on page 3.

Evidence that comes back PASS or FLAG joins the claim pool: the checked evidence your drafts may cite. A FAIL never enters it.

The second answer cites equation (3), and that FAILs too. An equation passes only where refereekit found its label printed beside a displayed equation. This paper numbers none. That FAIL is no evidence that the paper lacks the equation.

After the questions, a real review asks for your verdict and your answers to the editor. The tutorial answers them from a script. Your verdict is an input to the report, and the editor letter never sees it.

3. The triage list

The triage list gathers what needs you. Open it with /refx-flags in pi, or in the window’s “What needs attention” part:

NEEDS ATTENTION
citation errors — the cited page or the quoted words do not match the manuscript (1)
  absent — the quoted words are not in the manuscript (1)
    quotation (3) [q1] "words that are not on that page" — not found on page 3
extraction uncertainty — extraction could not vouch for the label; that is not proof the paper lacks it (1)
  equation-unresolved — extraction found no displayed equation with exactly this label (1)
    equation (3) [q2] — equation (3) is unresolved: the label was seen on pages 1, 3 and 9 but extraction did not accept it as a displayed equation's label (graphics-beside-label, no-column, no-display, no-math, not-at-column-edge, not-centred, page-margin, tabular, text-after-label); this is no evidence that the paper lacks it
verified: 1 PASS, 1 FLAG

It groups each problem by kind and reason, and says which question it came from. The last line counts what did verify. The equation FAIL sits under extraction uncertainty, because it is no proof that the paper lacks the equation.

4. What the agent learned

Ask the agent where the tutorial session stands. It learns the counts: two questions answered, with one PASS, one FLAG and two FAILs. It learns the claim pool size, two items, and the next step, to edit the drafts. It learns no question, answer, verdict or draft.

The session itself sits in your reviews folder, under work/tutorial/. In the agents with hooks, the hook guard refuses the agent’s ordinary tools there. It is not a sandbox. You can open it with your file manager or a text editor.

5. The drafts

Open the report with /refx-drafts in pi, or in the window’s Drafts part. It cites only what is in the claim pool:

The construction starts from "a finite set of nodes" (p. 1). Page 2 sets up the model. This is practice text written for the tutorial, not a review of the paper.

Edit it, in pi or in your own editor. Then check it. pi checks a draft each time you edit it, and the window has “Check it against the paper”:

checked against the original bound to this session (pdf sha256 9d0fc72513a2)
  PASS  p. 1 "a finite set of nodes": found on page 1 [found; pool: matching-pool-quotation]
  FLAG  p. 2: page 2 exists; no quotation to verify: 0 words, need 4 [no-quotation; pool: anchor-only] -- wording unchecked
2 citation(s): 1 PASS, 1 FLAG, 0 FAIL; 0 unanchored
not verified: citation forms other than 'p. N' and 'Eq. (N)', quotations under the floors, and every unquoted claim
exit 0 means no deterministic FAIL was found, not that the text is correct

An anchor is the page, equation or figure a claim points to. The label pool: anchor-only means the claim pool holds page 2, and the draft quotes no words there to match.

The agent now calls the report edited. That means only that the file changed. A later draft never replaces your edits without your yes.

6. Your own questions

The tutorial answers from a script, so it cannot take your questions. A question of your own needs a model backend, which you set in Model access (Install, part 2). With one, a question you type in the private view comes back the same way: statements, their evidence, and what each check found.

7. The Q&A page

The session’s Q&A page holds every question with its answer. The window links it as “The session’s review page”. There it loads nothing, so mathematics stays as raw TeX. Open the file in your browser yourself, and the page typesets the mathematics with MathJax, loaded from the web. That is the only request that leaves your machine in the tutorial, and your browser makes it.