refereekit

Version 0.2.0, unreleasedMIT licensePython 3.11 or later

Models quote.Code verifies.Referees judge.

refereekit checks every page, quotation and equation your referee report cites against the submitted PDF, before you send it.

refx_check1 PASS1 FLAG1 FAIL

The authors study “the realizability of simplicial complexes with prescribed degree-size sequences” (p. 1)PASS. The constraint on the joint sequence is set up on page 1FLAG, and the rest of the argument rests on it. The bound in Eq. (7)FAIL does not follow from that constraint as stated.

  1. PASSQuotation on p. 1found on page 1
  2. FLAGPage 1, nothing quotedpage 1 exists; no quotation to verify: 0 words, need 4
  3. FAILEq. (7)equation (7) is unresolved: extraction found no candidate with this label; this is no evidence that the paper lacks it
An example report, checked against refereekit's own test paper. A FLAG means the page exists and the wording was not checked. A FAIL is kept out of the claims your draft is written from.

Synopsis

refereekit is a toolkit for refereeing a paper with a language model, inside your agent: pi, Claude Code, Codex, Claude Desktop or the ChatGPT desktop app. You ask the model questions about the submitted paper, from a ready-made list or your own. It answers with quotations and page numbers, and refereekit checks every one against the PDF. The model then drafts the report and the editor letter in your voice, from the claims that verified.

It does not write your review. The verdict is yours. You give it before any drafting starts, and the draft is yours to edit.

Confidentiality comes first. A manuscript goes only to a model backend you have attested keeps nothing. It never goes into a repository, and never into your agent's context: the paper's words, and yours, appear only in a private view on your own screen. You never type a refereekit command, setup included.

Installation

You never type a refereekit command. The one prerequisite you install yourself is uv, from its own installer. Then paste the setup prompt into an agent that can run commands, such as pi, Claude Code or Codex. The agent checks for uv, installs refereekit, makes a reviews folder for the app you review in and checks it, each step under its own permission prompt. It asks you for no key and no password. The prompt begins:

Please set up refereekit on this computer for me. refereekit lets me referee a paper confidentially inside an AI agent like you. Work through the steps below in order. …

The whole prompt, and what each agent needs next, is in the install guide. Then ask your agent for the practice review. It works offline, needs no key and sends nothing. One runtime dependency, PyMuPDF. To review a real paper you set model access in a private view, never in the chat, and an OpenReview login if the venue uses OpenReview.

Compatibility

Language model
Needed for a real review: it answers your questions and writes the drafts. You choose the backend and attest that it keeps nothing, in a private view, never in the chat. The practice review needs none.
Coding agent

Required. You run refereekit by chatting with one of these agents. Its model calls refereekit's tools and never sees the paper.

AgentMinimumStatus
Claude Code2.1.295not yet qualified
Codex0.159.2not yet qualified
pi0.99.1not yet qualified
Claude Desktopnone yetnot yet qualified; no hooks
ChatGPT desktop appnone yetpreview; no hooks
Platforms
Ubuntu 24.04 on x86-64, and macOS 14 or newer on Apple silicon. Elsewhere refereekit installs, but no guarded agent integration, and so no review.
Python
3.11 or later.
Venues
Journals, and conferences on OpenReview.

Workflow

How each step runs in each agent, once its integration is installed in your reviews folder. The paper's words, and yours, stay in the agent's private view.

  1. setup

    Check that refereekit and this reviews folder are ready.

    Claude Code prompt/refx-setup
    Codex prompt$refx-setup
    pi: ask in plain wordsrefx_open
  2. tutorial

    Run the offline practice review of the public paper shipped with refereekit.

    Claude Code prompt/refx-tutorial
    Codex prompt$refx-tutorial
    pi prompt/refx-tutorial
  3. start

    Open a new review session on a PDF the referee names.

    Claude Code prompt/refx-start
    Codex prompt$refx-start
    pi: ask in plain wordsrefx_start
  4. status

    Show where a review stands from its records, without reading its files.

    Claude Code prompt/refx-status
    Codex prompt$refx-status
    pi: ask in plain wordsrefx_status
  5. draft

    Generate the report or the editor letter through the referee's model backend.

    Claude Code prompt/refx-draft
    Codex prompt$refx-draft
    pi: ask in plain wordsrefx_draft
  6. check

    Check the citations in a draft against the session's paper.

    Claude Code prompt/refx-check
    Codex prompt$refx-check
    pi: ask in plain wordsrefx_check
  7. finish

    Run the last deterministic checks before the referee submits by hand.

    Claude Code prompt/refx-finish
    Codex prompt$refx-finish
    pi: ask in plain wordsrefx_finish
  8. delete-session

    Delete one finished review session, after a dry run and the referee's explicit confirmation.

    Claude Code prompt/refx-delete-session
    Codex prompt$refx-delete-session
    pi: ask in plain wordsrefx_close

The setup prompt installs each agent's integration in your reviews folder. In Claude Desktop and the ChatGPT desktop app, ask in plain words; the agent calls the same tools. 6 more workflows come with the full profile: help, questions, voice, venue, recheck, rebuttal. Using a coding agent

If you use refereekit in your research, please cite the paper.

@inproceedings{yen2026models,
    title={Models Quote, Code Verifies, Referees Judge: When the Verifier Is the Weak Link},
    author={Tzu-Chi Yen},
    booktitle={NeurIPS 2026 Workshop on AI-Native Academia},
    year={2026},
    url={https://openreview.net/forum?id=k3FgymBG6u}
}