Reviewing on OpenReview

This page follows a submission assigned to you on OpenReview. You start a session on it, fetch the venue’s review form and the discussion, ask your questions, and draft the form’s prose fields. Then you read what came back from the authors. You do it all by chatting with your agent.

The steps that talk to OpenReview were not run while this page was written, because they need an account. What they do is read from the code. The drafting examples come from an offline session built from refereekit’s own test files.

Setup

OpenReview support is an optional part of refereekit. Ask your agent to add it, and the agent shows you the install before it runs it. Only starting a session from a submission number and fetching need it, and your login. Drafting the form and reading the replies work from what was already downloaded.

Enter your OpenReview user name and password in the “OpenReview account” part of the reviews folder’s page, which the agent opens for you. In pi, choose it in /refx-model. Never give them to the agent in the chat.

Give the agent the venue’s OpenReview ID, the full path form, such as ICLR.cc/2026/Conference. A wrong ID, a paper not assigned to you and a paper that does not exist all refuse with openreview-failed. OpenReview hides papers from anyone not assigned to them, so the last two look the same.

Find your assignments

Ask the agent which papers you are assigned at the venue. It learns how many there are and their submission numbers, never their titles, because a title is the paper’s own words. Read the titles on OpenReview itself. The answer also counts assignments that would not open, such as a withdrawn paper, so a short list shows as short.

The steps

Start. Ask the agent to start a review of submission 42 at ICLR.cc/2026/Conference. refereekit downloads the PDF, opens a session on it and records the venue and the number.
Fetch. Ask the agent to fetch the form and the discussion.
Review. Ask your questions and give your verdict in your private view, as for a journal.
Draft the form. Open the OpenReview part of your private view: “OpenReview form” in pi’s /refx-drafts, or in the review window. You may give each prose field a length there.
Read what came back. Ask the agent to analyze the authors’ replies.

Everything on Reviewing for a journal about the run, the flags and the drafts applies here too.

What a fetch brings

A fetch stores the venue’s review form and one file for each reply in the discussion. Replies go under the session’s theirs/ folder, which holds what others sent you and is never rewritten. A reply the authors revise arrives as a second file, so you can see what changed.

Neither part is required. Before the review stage opens there is no form yet, but you can start reading the paper. Before the discussion opens there are no replies. Fetch again later, and the fetch picks up what was missing.

Read the held count. The agent learns how many replies were new and how many were held back. A reply is held when refereekit cannot tell whether you wrote it. Your own review must not count as a reply, or the analysis would read it as agreeing with itself. Open the forum on OpenReview and check each held reply yourself. Nothing is lost, and nothing is assumed.

A download that is not a real PDF is refused, and leaves no paper in the session.

One session, one paper

One session holds one submission. Ask for submission 43 in a session that holds 42, and the fetch refuses before it writes anything. Start a new session for each submission.

Fetching the same number again is how you pick up a rebuttal or a revised PDF. A revision is stored beside the original. The claim pool, and every draft, keep checking against the original. Ask the agent to compare the session with the revision to see how your claims fare (see Comparing a revision). To review the revision itself, start a fresh session.

Drafting the form’s prose fields

Drafting needs the form, which only a fetch brings, and a claim pool and verdict from your review. Without them it refuses rather than invent the fields. It checks the venue, the form and your lengths before it sends anything, so a typo in a field name is reported as a typo.

On the offline session, the agent learned that two prose fields were drafted, two were left for you, and one citation was flagged. The draft cited page 3, which the questions never established. That flag means here what it means in a journal review (see Read the flags).

You read the draft in ours/openreview.md, in the private view or any text editor, and paste it into the web form yourself:

# Test.cc/2027/Conference/Submission42/-/Official_Review

## title
<!-- Brief summary of your review. -->

On p. 1 the paper says "a finite set of nodes". Page 2 sets up the model. On p. 3 it says "words that are not on that page".

## review
<!-- Please provide an evaluation of the quality, clarity, originality and significance of this work. -->

On p. 1 the paper says "a finite set of nodes". Page 2 sets up the model. On p. 3 it says "words that are not on that page".

## rating

(fill in yourself. options: 10: 10: Top 5% of accepted papers, seminal paper; 8: 8: Top 50% of accepted papers, clear accept; 5: 5: Marginally below acceptance threshold; 1: 1: Trivial or wrong)

## confidence

(fill in yourself. options: 5: 5: The reviewer is absolutely certain; 3: 3: The reviewer is fairly confident; 1: 1: The reviewer's evaluation is an educated guess)

Each field comes in the venue’s own order, with the venue’s instruction to you as a comment. The test model wrote the same prose under both text fields.

What gets drafted, and what never does

A field with a fixed set of choices is never filled in. A check can confirm that a quoted phrase is on a page. It cannot tell a soundness score of 3 from a 4. Ratings, scores and checkboxes always come back blank, for you.

Every free-text field is drafted, whatever the venue calls it. On a form shaped like ICLR’s, that includes the comment shown only to the area chairs. A box that asks you to disclose your use of language models, or to record your own unaided view, is free text too. So it would be drafted the same way. Rewrite those boxes by hand. A disclosure must be your own account of what you did, and a drafted one is false in exactly the way the box exists to prevent. Check what your venue requires before you fetch (see Before you start).

Drafting again is safe until you edit. Once you have edited the draft, a new one asks you first whether to replace it, and keeps your copy.

Reading what came back

The analysis reads every reply as what came back from others, and your drafted form as your own review. It writes ours/response-analysis.txt, which you read in any text editor.

It is a reading aid, not a verdict. It lists the points the response addresses, the points it does not, and the claims about the paper you should check again. It never recommends a rating or a decision. Author replies quote the paper, so this step sends the paper’s text to your model backend, behind the same gate as drafting.

The analysis always ends with this line:

NOTE: claims about a revised manuscript cannot be verified against doc.json, which holds the version originally fetched.

Nothing in a reply has been checked against the paper. An author who describes a change is describing a version refereekit has not checked. Fetch again to store the revision, and compare the session with it.

Read-only

refereekit never posts to OpenReview. There is no code that could. You submit a review yourself, in your browser, having read what you submit.

Confidentiality

A fetched paper, its extracted text and every draft made from it are confidential. Keep every session under your reviews folder’s work/, where refereekit puts it (see Repository rules). The form itself is the venue’s public configuration, and holds no paper text.

The venue gate runs on the venue the session recorded. A venue known to forbid outside models stops the review, the form’s drafting and the analysis, with nothing for you to repeat.