Welcome to Review Quality Collector (RQC)

Elsewhere:

• RQC overview

1. What is Review Quality Collector (RQC)?

Review Quality Collector (RQC) is an initiative for improving the quality of scientific peer review. Its core is a mechanism that supplies a reviewer with a receipt for their work for each conference instance or journal year.

2. How does RQC work?

Very roughly like so:

• The PC chairs of a conference or the editors of a journal decide to use RQC and configure an RQC instance.
• They agree on a review quality definition (RQdef) and upload it to RQC. An RQdef defines the relative weight of the four aspects "Helpfulness for Authors", "Helpfulness for Decision", "Timeliness", and "Has Graded Co-Reviews" and the few or many facets that define the first two of these. This is best explained by an example (PDF).
• Each reviewer grades their co-reviewers' reviews and/or journal editors grade the reviews. Reviewers can opt out of RQC. (The reviewing data comes from the usual manuscript handling system.)
• RQC computes a ranking of the reviewers and sends each reviewer that has not opted out their detailed receipt. Again, this is best explained by an example (PDF). (This is the "RQ" part of RQC.)
• Reviewers can assign their receipts to a particular department within some research institution and those institutions will receive the collected receipts of their researchers without effort. (This is the "C" part of RQC, not implemented yet.)

3. Why does RQC help?

RQC imagines three sorts of reviewers:

• One group will always provide good reviews, because they know it is important.
• One group will rarely or never provide good reviews, because they lack the skill.
• In between is the largest group: reviewers who could do better reviews, but too many things that promise more reputation limit their motivation. RQC will improve their motivation by turning reviewing into a direct and reliable source of reputation.

4. Advantages of RQC

4.1 Advantages for reviewers

• Reviewers will finally get something to show for their review effort. The receipt is a tangible source of reputation.
• Reviewers get feedback regarding what's good in their reviewing and what is not that can serve as a starting point for improvement.

4.2 Advantages for authors

• Authors will more often receive reviews that are timely and helpful.
• Negative reviews will less frequently use an impolite (or worse) tone.

4.3 Advantages for editors and conference chairs

• Reviews will less frequently come late.
• Editors and chairs can use the review quality framework to express what they are and are not looking for in reviews.
• More reviewers will spend the necessary effort to provide a solid review.
• The receipt acts as a payment for the reviewing effort in the currency that counts in research: reputation.
• Journals and conferences can praise their best reviewers based on solid data.

4.4 Advantages for research institutions

• Research institutions can more formally recognize reviewing as an important researcher activity.
• They can use reviewing quality and quantity as elements of researcher performance evaluation.

4.5 Advantages for publishers

• Publishers can maintain the willingness of their reviewers to provide reviews for them.
• Over time, reviewers will gravitate towards those journals where they can provide the most useful input.

5. Guiding principles

RQC is constructed according to four key ideas:

5.1 Transparency

RQC will provide transparency at the level of journal, conference, research institution, and publisher. It allows individual reviewers to provide additional transparency at the individual level.

5.2 Simplicity

RQC will not provide every imaginable bell and whistle. Instead, it emphasizes simplicity in order to stay understandable and reliable.

5.3 Privacy

At the level of the individual, RQC gives reviewers control which of their receipts to make public or show to individual people or groups. RQC will not make other use of private data except in aggregate or otherwise anonymized form or where explicitly allowed by the user.

5.4 Literacy

RQC is a service for people who not only can read, but actually do read.

6. What one may dislike about RQC

• Some people dislike having yet one more quantification of their work.
• Some people dislike the fact that in a percentile-based ranking, a large minority of participants will be below average.
• Grading reviews is a non-trivial additional work step.

7. Implementation status

• Receipts can be issued for conferences run on EasyChair or HotCRP.
• RQC-for-journals and its web API can be tried out in demomode
• First manuscript handling system adapter for RQC (for OJS) is available.
• Full-fledged, non-demomode RQC-for-journals is mostly finished and will be released soon.

See the past and future for more detail on the current and coming implementation status, the how-to for learning how to use RQC, or the overview for everything.