Your first review, start to finish

This page takes you from nothing installed to a submitted report and a deleted session, in one chat with your agent. Each step gives a line to say, and what the agent does. The practice run needs no account and no key.

Words used here

Agent: the program you chat with, such as pi or Claude Code. Its model calls refereekit’s tools. refereekit never gives the paper to the agent’s model.
Reviews folder: one folder on your computer. It holds your refereekit settings and, under work/, one private directory per review.
Session: one review of one paper. It is named by a generated reference such as s-1a2b3c4d, never by the paper’s title.
Model backend: the language model that reads the manuscript, answers your questions and drafts. You choose it, and you attest that it keeps nothing.
Private view: where the paper’s words and yours appear. In pi it is pi’s own dialogs and editor. In the other agents it is the review window, a page on your own machine.

The lines below are for pi and Claude Code. Codex writes each command with $ where Claude Code uses /. In Claude Desktop and the ChatGPT desktop app, ask in plain words. The guides has each spelling.

1. Install

Paste the setup prompt from Install into your agent. It is one long message that starts:

Say this: Please set up refereekit on this computer for me. …

The agent checks for uv, then installs refereekit. Approve each command in the agent’s permission prompt. The agent never asks you for a key or a password.

2. Create a reviews folder

The agent asks where to keep your reviews:

Agent: Where should I keep your reviews? For example, a new folder named reviews in your home folder.

Say this: Yes, ~/reviews.

The agent makes the folder, then checks that the practice run can go ahead. It reports whether each setting is set, never its value. Edit STYLE.md, the voice of your drafts, yourself or through the agent. The files you edit lists yours.

3. Restart in the folder

The agent’s last setup message says what to restart or trust. pi loads refereekit when it starts again in the reviews folder. Claude Code asks you to approve the folder’s refereekit server. Codex asks you to trust the folder and to approve its hooks with /hooks. Say yes, or the tools are missing. Claude Desktop and the ChatGPT desktop app load it when you quit and open them again.

4. Practice on a public paper

Say this: /refx-tutorial

The tutorial reviews a public paper that ships with refereekit. Its questions and answers are fixed, and no model runs. Each answer appears with its checks in your private view. The tutorial shows the whole run. The first answer is written to show all three verdicts:

PASS: the quoted words occur on the named page, and nothing more. It does not say the sentence around them is true, or fair to the paper.
FLAG: the page exists, but fewer than four words were quoted, so its wording is unchecked. Read it yourself.
FAIL: the words are not on that page, or the page does not exist.

Evidence that comes back PASS or FLAG joins the claim pool. The claim pool is the set of checked evidence your drafts may cite. A FAIL never enters it.

A fourth count also appears. UNANCHORED means a quotation that names no page, so it was not checked. It is neither a PASS nor a FAIL.

The second answer cites equation (3), and that FAILs too. An equation passes only where refereekit found its label printed beside a displayed equation, and this paper numbers none. A FAIL like this is no evidence that the paper lacks the equation.

Run it as often as you like, then ask:

Say this: Where does the tutorial session stand?

The agent learns the session’s counts and nothing else. Here it learns that two questions were answered with one PASS, one FLAG and two FAILs. The claim pool holds two items, a verdict is on record, and the next step is to edit the drafts. The agent learns no question, no answer and no draft.

5. Set model access

A real manuscript goes only to a model backend you chose. You set it in Model access, a private view:

Say this: /refx-model

That is pi. In the other agents:

Say this: Open model access for me.

Agent: I opened the reviews folder’s page in your browser. Choose the backend and enter its settings there. I will not see them.

Choose the deployment and enter its settings. Tick the attestation only if the account behind the model backend keeps nothing. That is your statement, and no step can verify it. Never paste a key or a password into the chat. Install, part 2 says what each deployment asks for.

Then check that you are ready:

Say this: Is refereekit ready for a real review?

The agent runs a check that sends nothing and reports each result. A failed check carries a code, and Troubleshooting says what to do. Account retention always stays unknown: no check can see your account.

refereekit refuses a venue only if it is on its list, which holds one venue, NeurIPS. If yours forbids sending a submission to a language model, add it to venue-policy.toml in the reviews folder. Then name the venue when you start the review. The gate acts only on a listed venue you named. See the venue gate.

6. Start the review

Ask the agent to start a review of the PDF:

Say this: /refx-start Start a review of ~/Downloads/paper.pdf for PRX.

In pi, ask in plain words. The agent learns the session’s reference, its page count, whether the venue permits a model, and the next step: ask your questions.

The session is bound to this PDF. A different PDF is refused, and a revision can sit beside it for comparison. refereekit never gives its text to the agent’s model. If you name no venue, the review asks for it.

The path and the venue stay in the chat. Keep the paper’s title out of both. See What you give the agent.

7. Ask your questions in the private view

Say this: /refx-review

That is pi. In the other agents, the review window opens at the question form.

Type a question in your own words, or choose one from your question list. The model answers in statements, and links each to its evidence. refereekit checks every piece of evidence at once, and shows the answer with its checks. Every question is recorded. In pi, a blank question ends the loop.

Then give your verdict: your recommendation, the venue, and whether the revision is major or minor. The verdict is yours. It is an input to the report, never an output of it. Then answer the editor’s questions, if any. The report and the editor letter are drafted from the claim pool.

In pi, /refx-review asks for each of these in turn and shows each draft. In the window, record the verdict under “Your verdict”. Then use “Draft the report” and “Draft the letter”.

8. Edit the drafts

Say this: /refx-drafts

In pi, this opens a draft in an editor. In the window, “Open it in your editor” does the same. Make the drafts yours. The agent then calls them edited, which means only that the file changed. It does not mean anyone read or approved them. A later draft never overwrites your edits without your yes in the private view, and your version is kept.

A co-referee’s report is checked in the same place, under “Check a file”.

9. Check, then finish

Say this: /refx-check, then /refx-finish.

The check runs over your edited drafts. Each citation in a form the check reads (p. N, page N, Eq. (N)) is checked again. Other forms are not; the drafts view lists most of them. The finishing checks then ask three questions. Do both drafts exist? Does any citation FAIL? Are the form fields you must fill filled in?

The agent learns only the counts. To read the details, ask the agent to open the finish view. On the tutorial session it shows:

finish: checked against the original document bound to this session
  report: 1 PASS, 1 FLAG, 0 FAIL, 0 UNANCHORED
    FLAG p. 2: page 2 exists; no quotation to verify: 0 words, need 4
    still exactly as generated
  editor: 0 PASS, 1 FLAG, 0 FAIL, 0 UNANCHORED
    FLAG p. 1: page 1 exists; no quotation to verify: 0 words, need 4
    still exactly as generated
still yours to judge: every FLAG (page exists, wording unchecked), every UNANCHORED quotation, and every claim not in quotation marks
exit 0 means nothing mechanical is missing or FAILs; it is not an approval
submitting is yours: paste the drafts into the venue's own system. refereekit submits nothing.

A clean finish means nothing is missing and nothing FAILs. It is not approval. It cannot tell whether your argument is right.

10. Submit

Paste the drafts into the venue’s system yourself. refereekit cannot submit anything.

11. Delete the session

When the review is over, and the venue’s terms allow it, delete the session:

Say this: /refx-delete-session s-1a2b3c4d

In pi, ask in plain words. The agent shows you what deleting destroys. Then it waits for your yes on your own screen, in a dialog or a button. A reply in the chat does not count.

Deleting destroys the session’s copy of the PDF and its extracted text. It also destroys the claim pool, the questions and answers, every draft and any received reports. Your venue notes stay. The original PDF you started from is not touched, so delete it yourself.