refereekit

refereekit

Changelog

Unreleased

0.2.0

Not yet published. Agent integration is built and tested offline but not yet qualified against the real agents; see internal/notes/release-checklist.md.

The review in your agent

  • BREAKING: the human command line is gone. review, serve, verify, mem-store, mem-recall, check-answer, evidence and every human view are removed and exit 2 as unknown commands. Every command that remains is plumbing that answers in JSON, in any shell; --json is accepted and ignored for one release.
  • The review now runs in pi, Claude Code or Codex. You chat with the agent; its model calls the refx_* tools and gets counts and codes, never the paper.
  • pi gets an extension with /refx-review, /refx-drafts, /refx-flags, /refx-note and /refx-tutorial. The paper's words and yours appear only in pi's own interface, which pi does not save to its session files.
  • Claude Code and Codex get refereekit mcp, a local MCP server. The paper's words and yours appear only in a review window on 127.0.0.1, behind a token that changes per window, and drafts open in your own editor. The window is weaker than pi's view; the docs say how.
  • Deleting a session asks for your yes on your own screen, never in the chat: in pi's confirmation dialog, the client's confirmation form, or the review window's button.

No terminal for referees

  • You never type a refereekit command, setup included. The docs give one setup prompt that you paste into an agent that can run commands; the agent checks for uv and the host, installs refereekit with its model and OpenReview support, makes the reviews folder and runs doctor, each step under its own permission prompt. uv itself is the one prerequisite you install, from its own installer.
  • Model access is set in a private view: /refx-model in pi, or the reviews folder's page (refx_open with the view setup) in the other agents. It takes the deployment, its settings and the attestation, and writes the reviews folder's .env, owner-only. A stored key shows as set, never by its value.
  • refereekit reads the names it uses from the reviews folder's .env after the process environment, which wins, so a desktop app's server needs no shell. A .env outside a folder holding refereekit.toml is never read. No tool takes a credential or the attestation as an argument. The hooks already refused the agent any access to .env; tests now pin that writes are refused too.
  • Claude Desktop and the ChatGPT desktop app (a preview) are hosts, with the same tools and the same review window. They run no hooks, and doctor reports them as unguarded.
  • Every step the hook keeps for you has a private-view home: "Check a file" and an OpenReview form part in the drafts view, and refx_setup for removing or reinstalling the integration on your yes. A referee-only refusal names that step.
  • The published docs, the README and the landing page show no command for the reader; a test enforces it. The command reference is now docs/reference/tools.md, with a pointer at the old address.

Verification

  • A session is bound to the PDF it was started from; ingest, the review and or-fetch never replace it, and a revised submission is stored beside it.
  • An equation citation passes only when ingest recorded that label beside a displayed equation. A label it could not vouch for fails as unresolved, which is no finding that the paper lacks the equation.
  • pi's /refx-flags draws an equation label's recorded region from the source PDF, so you can see what its PASS rests on.
  • checks lists every kind of evidence refereekit can check, what each needs, and what its PASS establishes. A citation of any other kind is refused rather than given a verdict.
  • check verifies the page and equation citations in a referee's own text against the bound PDF, and reports what it could not check.
  • recheck compares the claim pool with a revision, scoped to extracted text.

Questions

  • Each question asks the model for a structured answer: statements, each linking the evidence it rests on, such as a quotation with its page. Every linked record is checked, and an answer that breaks the structure is reported, never cited. No summary is asked for unless you ask.
  • Every question is recorded before it is sent; resumed sessions continue the numbering.
  • Shared questions.toml names generic templates only (refereekit questions); the question loop offers them to pick from.
  • New experiment runs and rescores are named and never write into the submitted record set.

Lifecycle

  • init, start, status, doctor, the tutorial, finish and close; sessions named by generated references and addressed with --session-ref.
  • Draft generations: an edited draft is never overwritten unless you say so, and the old bytes are kept.

Agents

  • init --tools claude,codex,pi, update, uninit, instructions and the guard hook entry point; eight core refx-* workflows, six more in the full profile.
  • Every answer, from a command or a tool, is validated against a fixed schema.
  • A venue refusal now shows, in the private view, an override that actually lifts it.
  • pi's ! commands go through the hook guard too, the pi guard refuses in a form pi shows and no longer freezes pi, and the Claude Code hook blocks when it cannot run (Claude Code 2.1.295 or newer).
  • The hook guard refuses the agent the commands that are yours, as referee-only: uninit, init in an existing reviews folder, close --yes, check of a file, and every command that carries your words (draft --length, editor --answers, or-draft --length).