How to organize peer review for a conference

The size of your reviewer pool follows from three numbers you control. Work it out before you recruit anyone:

Change any input and the pool changes with it. If you cannot recruit the number the arithmetic asks for, reduce reviews per abstract for one submission type, or extend the review window — not the reviewers’ load.

1. 1

   Choose the review model

   Single-blind, double-blind or open. Double-blind protects against judging the author rather than the work; it also means telling authors what to leave out. Single-blind vs double-blind review compares them.

2. 2

   Agree the criteria and the scale

   Pick a few criteria — scientific quality, originality, relevance and methodology are a common set — and describe what each score means. A written rubric is what makes one reviewer’s 4 comparable with another’s. The scoring rubric template is a starting point.

3. 3

   Build the reviewer pool

   Invite reviewers with their institution and a short list of topics. Topics drive assignment; institutions drive conflict checks. Recruit from the numbers you planned.

4. 4

   Set conflict-of-interest rules

   At minimum: nobody reviews their own work, and nobody reviews work from their own institution. Let reviewers declare conflicts you cannot see, such as a recent collaborator.

5. 5

   Assign the submissions

   Match topics to tracks and keywords, spread the load evenly, and check every pairing against the conflict rules. Assign each abstract the same number of reviewers so averages compare fairly.

6. 6

   Set a deadline and follow up

   Give one firm date. A week before it, remind reviewers with outstanding assignments, and have a small reserve of reviewers ready for the ones who decline.

7. 7

   Decide as a committee

   Read the scores as evidence, not a verdict. Look first at submissions where reviewers disagree — a 5 and a 1 average to a 3 that describes neither — then settle clear accepts and rejects quickly.

8. 8

   Record the decisions, then publish the accepted work

   Record accept, reject or request revision for every submission, with a note where it helps. Publish the accepted abstracts only when the list is final, then send decisions to authors.

Disagreement is information. A wide spread between reviewers usually means one of three things: the abstract is outside one reviewer’s field, the criteria were read differently, or the work is genuinely divisive. The first calls for another reviewer, the second for a clearer rubric next year, the third for a committee discussion. None of them is resolved by averaging.

• Tell authors, in the call, to keep names, institutions and identifying self-references out of the title and abstract.
• Withhold author names and affiliations from reviewers’ data, not just from their screen.
• Keep the committee’s view separate: the people deciding can see authors; the people scoring cannot.

Choose single blind, double blind (the default) or open review; set one to ten reviewers per abstract (three by default), a deadline, and up to ten criteria scored 1 to 5. Reviewers are invited by email and see only their own assignments. Assignment can be manual, by a fixed topic-and-load rule, or from AI suggestions you confirm — and the conflict rules apply to all three. The decisions screen shows every review, the average, each recommendation and an AI summary, with the committee making every decision. Details are on conference peer review software.