Reviewer Guidelines

Thank you for agreeing to review for CVPR 2019. To maintain a high-quality technical program, we rely very much on the time and expertise of our reviewers. This document explains what is expected of all members of the Reviewing Committee for CVPR 2019.

The CVPR 2019 Reviewing Timeline

Paper Submission Deadline November 16, 2018 [11:59 p.m. Pacific Standard Time]
Supplementary materials November 23, 2018 [11:59 p.m. Pacific Standard Time]
Reviews Released to Authors February 1, 2019 [Pacific Standard Time]
Rebuttal Period February 1-8, 2019 [Pacific Standard Time]
Final Decisions to Authors March 2, 2019 [Pacific Standard Time]

The CVPR 2019 Reviewer Guidelines

Read this CVPR 2019 Reviewer Tutorial for a summary of the decision process, annotated good/bad reviews, and tips. You may also be interested in this CVPR 2019 Area Chair Tutorial that advises on how to select reviewers for papers and make decisions.

Blind Reviews

Authors were asked to take reasonable efforts to hide their identities, including not listing their names or affiliations and omitting acknowledgments. This information will of course be included in the published version. Reviewers should also make all efforts to keep their identity invisible to the authors. Please see the Author Guidelines for details on how authors have been instructed to preserve anonymity, including guidelines for referencing one's own prior work. We realize that with the increase in popularity of publishing technical reports and ArXiv papers, sometimes the authors of a paper may be known to the reviewer. As voted on at the CVPR 2015 PAMI-TC meeting, ArXiv papers are not considered prior work since they have not been peer reviewed. Therefore, you should review your CVPR papers independently as if the ArXiv papers didn't exist. Citations to these papers are not required and failing to cite or beat performance of arXiv papers are not grounds for rejection. Please read the FAQ below for guidelines on handling ArXiv papers. An important general guideline is to make every effort to treat papers fairly whether or not they know (or suspect) who wrote them. There are some specific examples in the Reviewer FAQs below. Reviewers should not search for the authors of a paper, and complain that the paper is not anonymous if they happen to find them.

Check your papers

As soon as you get your reviewing assignment, please go through all the papers to make sure that (a) there is no obvious conflict with you (e.g., a paper authored by your recent collaborator from a different institution) and (b) you feel comfortable to review the paper assigned. If either of these issues arise, please let us know right away by emailing the Program Chairs.

What to Look For

Look for what's good or stimulating in the paper. Minor flaws can be corrected and shouldn't be a reason to reject a paper. CVPR as a conference is looking for new ideas. We recommend that you embrace novel, brave concepts, even if they have not been tested on many datasets. For example, the fact that a proposed method does not exceed the state of the art accuracy on an existing benchmark dataset is not grounds for rejection by itself. Acceptance and rejection decisions should not be determined solely by the method's raw performance. Rather, it is important to weigh both the novelty and potential impact of the work alongside the reported performance. Each paper that is accepted should be technically sound and make a contribution to the field.

Be Specific

Please be specific and detailed in your reviews. In the discussion of related work and references, simply saying "this is well known" or "this has been common practice in the industry for years" is not sufficient: cite specific publications, including books, or public disclosures of techniques.
Your main critique of the paper should be written in terms of a list of strengths and weaknesses of the paper. Use bullet points here, and explain your arguments. Your discussion, sometimes more than your score, will help the authors, fellow reviewers, and Area Chairs understand the basis of your opinions, so please be thorough. Your reviews will be returned to the authors, so you should include specific feedback on ways the authors can improve their papers. For more suggestions on writing your reviews, read the section below on Writing Technical Reviews.

Be Careful

Please think carefully about your reviews. In particular, it's a good idea to avoid ad-hoc policy innovations, which can occur with the best of intentions. Here is an example. Author submits a paper relying on a dataset that cannot be public. Reviewer takes the position that this means the results cannot be trusted, and rejects on these grounds. The problem with this review is that it's clearly about a matter of CVPR policy, rather than about the paper's content. We have clear policies about double submission and plagiarism. CVPR doesn't have a policy about non-public datasets, and it's unfair for reviewers to invent one of their own.

Rebuttals

In your review, you may request clarifications or additional illustrations in the rebuttal. Per a passed 2018 PAMI-TC motion, reviewers should not request additional experiments for the rebuttal, or penalize for lack of additional experiments. Authors should not include new experimental results in the rebuttal, and you should discounted any new results when making your final recommendation.

When You're Done

When you have finished with your review, you should destroy any paper manuscript and/or supporting material you received. See the Ethics guidelines below.

Writing Technical Reviews

Here are some recommendations that may help you as you do this very valuable task.
We volunteer our time by reviewing papers that are written by other researchers in our field. We recommend that you approach your reviews in this spirit of volunteerism. Your reviews make you a gatekeeper in helping decide which papers are ready for publication. Just as important, however, is to provide feedback to the authors so that they may improve their work. Try to write your review in a way that the authors can benefit from. We suggest reading a paper and then thinking about it over the course of several days before you write your review.
The tone of your review is also important. A harshly written review will be disregarded by the authors, regardless of whether your criticisms are true. If you take care, it is always possible to word your review diplomatically while staying true to your thoughts about the paper. Put yourself in the mindset of writing to someone you wish to help, such as a respected colleague who wants your opinion on a concept or a project.
Here are some specific issues to keep in mind as you write your reviews:

  • Short reviews are unhelpful to authors, other reviewers, and Area Chairs. If you have agreed to review a paper, you should take enough time to write a thoughtful and detailed review.
  • Be specific when you suggest that the writing needs to be improved. If there is a particular section that is unclear, point it out and give suggestions for how it can be clarified.
  • Don't give away your identity by asking the authors to cite several of your own papers.
  • Be specific about novelty. Claims in a review that the submitted work "has been done before" MUST be backed up with specific references and an explanation of how closely they are related. At the same time, for a positive review, be sure to summarize what novel aspects are most interesting in the strengths.
  • Citations to papers that have only been published without review (e.g. ArXiv or Technical reports) are not required. Therefore, missing these citations is not grounds for rejecting a paper.
  • If you think the paper is out of scope for CVPR's subject areas, clearly explain why in the review. You may find the Call for Papers here. Then suggest other publication possibilities (journals, conferences, workshops) that would be a better match for the paper.
  • Avoid referring to the authors by using the phrase "you". These phrases should be replaced by "the authors" or "the paper." Referring to the authors as "you" can be perceived as being confrontational, even though you do not mean it this way.

Be generous about giving the authors new ideas for how they can improve their work. Your suggestions may be very specific (for example, "this numerical solver would be better for your application") or may be more general in nature. You might suggest a new dataset that could be tried, or a new application area that might benefit from their tool. You may tell them how their idea can be generalized beyond what they have already considered. A thoughtful review not only benefits the authors, but may benefit you as well. Remember that your reviews are read by other reviewers and especially the Area Chairs, in addition to the authors. Being a helpful reviewer will generate good will towards you in the research community.

Ethics for Reviewing Papers

1. Protect Ideas

As a reviewer for CVPR, you have the responsibility to protect the confidentiality of the ideas represented in the papers you review. CVPR submissions are not published documents. The work is considered new or proprietary by the authors; otherwise they would not have submitted it. Of course, their intent is to ultimately publish to the world, but most of the submitted papers will not appear in the CVPR proceedings. Thus, it is likely that the paper you have in your hands will be refined further and submitted to some other journal or conference, or even to CVPR next year. Sometimes the work is still considered confidential by the authors' employers. These organizations do not consider sending a paper to CVPR for review to constitute a public disclosure. Protection of the ideas in the papers you receive means:

  • You should not show the paper to anyone else, including colleagues or students, unless you have asked them to write a review, or to help with your review.
  • You should not show any results or videos/images or any of the supplementary material to non-reviewers.
  • You should not use ideas from papers you review to develop new ones.
  • After the review process, you should destroy all copies of papers and videos and erase any implementations you have written to evaluate the ideas in the papers, as well as any results of those implementations.
2. Avoid Conflict of Interest

As a reviewer of a CVPR paper, it is important for you to avoid any conflict of interest. There should be absolutely no question about the impartiality of any review. Thus, if you are assigned a paper where your review would create a possible conflict of interest, you should return the paper and not submit a review. Conflicts of interest include (but are not limited to) situations in which:

  • You work at the same institution as one of the authors.
  • You have been directly involved in the work and will be receiving credit in some way. If you're a member of the author's thesis committee, and the paper is about his or her thesis work, then you were involved.
  • You suspect that others might see a conflict of interest in your involvement.
  • You have collaborated with one of the authors in the past three years (more or less). Collaboration is usually defined as having written a paper or grant proposal together, although you should use your judgment.
  • You were the MS/PhD advisor of one of the authors or the MS/PhD advisee of one of the authors. Most funding agencies and publications typically consider advisees to represent a lifetime conflict of interest. CVPR has traditionally been more flexible than this, but you should think carefully before reviewing a paper you know to be written by a former advisee, especially a recent one.

While the organizers make every effort to avoid such conflicts in the review assignments, they may nonetheless occasionally arise. If you recognize the work or the author and feel it could present a conflict of interest, email the Program Chairs as soon as possible so he or she can find someone else to review it.

3. Be Professional

Belittling or sarcastic comments have no place in the reviewing process. The most valuable comments in a review are those that help the authors understand the shortcomings of their work and how they might improve it. Write a courteous, informative, incisive, and helpful review that you would be proud to add your name to (were it not anonymous).

Reviewer Instructions

The following provides further details for reviewing papers using the CVPR 2019 submission and reviewer system. In addition, please see the Reviewer Guidelines above and the FAQs below.

  • The submission/review site is https://cmt3.research.microsoft.com/CVPR2019 (bookmark or save this URL!).
  • Please make sure that your browser has cookies and Javascript enabled.
  • Please add "email@msr-cmt.org" to your list of safe senders in your own email client to prevent important email announcements from being blocked by spam filters.
  • Do not create a new account on CMT! If you need to change the email address on your account, email the PCs and we will do it for you. If you need to change only your name, you can do this in the CMT system yourself using the same account.

Once you've been notified that the papers have been assigned to you, please log in to the CMT site and follow these steps:

1. Download papers and check for possible conflict or submission rule violations:

In the Section "Paper Reviews and Discussions", click on "Download Assigned Papers". This allows you to download a zip file containing all the papers plus supplementary files (if available).
Contact the Program Chairs immediately if:

  1. You think you are in any way conflicted with the paper (see above).
  2. There is a violation of the stated paper submission rules. Such a violation includes:
    • Over 8 pages (not including references)
    • Constitutes a double submission (there is a detailed, clear and specific definition of a double submission in the instructions to authors; it is reproduced below in the reviewer FAQs)
    • Supplementary material includes a newer version of the paper
    • Plagiarism see the reviewer FAQs
    Please specify the exact nature of the violation. For your reference, the paper submission guidelines, which includes descriptions of CVPR 2019's dual submission policy, can be found in the Author Guidelines.

2. Familiarize yourself with the data under "Detailed Reviews and Discussions":

"Paper Summary" label: next to it, you'll see the icons "+" and "-". Clicking on "+" shows you all the abstracts; clicking on "-" collapses all them back.
At the end of each paper title, you'll see "+" as well. This has the same function of showing the abstract for that paper, toggling to "-" at the same time, which collapses it when selected.
Please take the time to familiarize yourself with the table entries; clicking on any of the column heading (e.g., "Paper ID") sorts according to its description.

3. Review papers and assign them a Preliminary Rating:

For a paper, under the review column, click "Add" (to the right of the "Review" line) to enter a review. Please see the Reviewer Guidelines above and take each review seriously. The review form contains detailed instructions about writing the review and the meaning of the different ratings (Oral, Poster, etc.).
CMT does not allow users to type certain characters into a text box that could be interpreted as html tags or a malicious script. As a workaround, introducing spaces between these characters (for example, "y < x") will allow you to submit the text since this can no longer be interpreted as an html tag.
If you save your review as a draft, it is visible only to you. You can access your draft review form by clicking on the same "Add" link. To make the review visible to the Area Chair, click on the "Submit" button in the review form. "Submit" won't work if any of the required items is not filled.

4. (Optional) Review papers offline:

To enable the "Offline Reviewing", you can use “Actions - > Import Reviews” menu from your reviewer console to review papers offline.
You can select papers and click 'Download' to obtain the review stubs to update files as needed. Once you are done updating, you can upload the file from the same page.
Please read instructions on how to modify the file to incorporate your responses. Note that you must not add certain characters in your responses that could be interpreted as html tags or a malicious script. See item 3 above.
You can upload the completed file using the "Upload" interface at the bottom of the page. The new uploaded version will (destructively) overwrite the current review.
We suggest that you try downloading a review template file for one paper, enter test responses, and upload to get a sense of how it works.
You should always verify the review after uploading (by inspecting it online).
We suggest that you use an XML editor to edit the file, for example: EditiX (Windows, Unix/Linux, Mac OS X) or XML Notepad 2007 (Windows only). (Remember to edit only fields currently filled with the phrase "REPLACE THIS WITH YOUR ANSWER".)

5. Participate in discussions with Area Chairs and other reviewers:

After the rebuttal period, reviewers will work with Area Chairs to clear up any confusions and attempt to reach consensus on papers.
The CMT site has an electronic bulletin board feature that allows Area Chairs to contact reviewers anonymously. Once the Area Chair posts a note, reviewers will be notified and asked to log in to see the post and respond. The identities of the reviewers will be hidden from each other.

6. Enter your Final Recommendation

After the rebuttal period you will enter your final recommendation on CMT. This may differ from your preliminary rating, and should reflect your judgment taking into account all the other reviews, the authors' rebuttal, and the discussion about the paper (if any).

Reviewer FAQs

Is there a minimum number of papers I should accept or reject?
No. Each paper should be evaluated in its own right. If you feel that most of the papers assigned to you have value, you should accept them. It is unlikely that most papers are bad enough to justify rejecting them all. However, if that is the case, provide clear and very specific comments in each review. Do NOT assume that your stack of papers necessarily should have the same acceptance rate as the entire conference ultimately will.

Can I review a paper I already saw on arXiv and hence know who the authors are?
Yes. See next bullet below for guidelines.

How should I treat papers for which I know the authors?
Reviewers should make every effort to treat each paper fairly, whether or not they know who wrote the paper. For example: It is Not OK for a reviewer to read a paper, think "I know who wrote this; it's on arXiv; they're usually quite good" and accept paper based on that reasoning. Conversely, it is also Not OK for a reviewer to read a paper, think "I know who wrote this; it's on arXiv; they're no good" and reject paper based on that reasoning.

How should I treat arXiv papers?
ArXiv papers are not considered prior work since they have not been peer reviewed. Therefore, you should review your CVPR papers independently as if the ArXiv papers didn't exist. Citations to these papers are not required and failing to cite or beat performance of arXiv papers are not grounds for rejection. For example:

  • It is Not OK for a reviewer to suggest rejection for not citing an arXiv paper or not being better than something on arXiv.
  • It is Not OK to accept a paper solely because it performs better than something on arXiv.
  • It is Not OK to reject a paper solely because it performs worse than something on arXiv.
  • It is Not OK to regard arXiv as a standard for the state of the art, because it is not reviewed. This applies *whoever* wrote the arXiv paper.
  • It is Not OK for a reviewer to reject a paper solely because another paper with a similar idea has already appeared on arXiv. If the reviewer is worried about plagiarism they should bring this up in confidential comments to the AC.
  • It is OK for a reviewer to suggest an author should acknowledge and be aware of something on arXiv.
  • It is OK for an author to decline to acknowledge something on arXiv (because it has not been reviewed and so may not be right).

Dual/Double Submissions: By submitting a manuscript to CVPR, authors acknowledge that it has not been previously published or accepted for publication in substantially similar form in any peer-reviewed venue including journal, conference, workshop, or archival forums. Furthermore, no paper substantially similar in content has been or will be submitted to another conference or workshop during the review period (November 16, 2018 - March 2, 2019). The authors also attest that they did not submit substantially similar submissions to CVPR 2019. Violation of any of these conditions will lead to rejection.

The goals of the dual submission policy are (i) to have exciting new work be published for the first time at CVPR, and (ii) to avoid duplicating the effort of reviewers.

Our policy is based upon the following particular definition of “publication”. A publication, for the purposes of the dual submission policy, is defined to be a written work longer than four pages that was submitted for review by peers for either acceptance or rejection, and, after review, was accepted. In particular, this definition of publication does not depend upon whether such an accepted written work appears in a formal proceedings or whether the organizers declare that such work “counts as a publication”.

Note that such a definition does not consider an arXiv.org paper as a publication because it cannot be rejected. It also excludes university technical reports which are typically not peer reviewed. However, this definition of publication does include peer-reviewed workshop papers, even if they do not appear in a proceedings, if their length is more than 4 pages including citations. Given this definition, any submission to CVPR should not have substantial overlap with prior publications or other concurrent submissions.

As a rule of thumb, the CVPR submission should contain no more than 20 percent of material from previous publications. Authors are encouraged to contact the Program Chairs about clarifications on borderline cases.

An extended version of a paper submitted to CVPR (with sufficiently new material) can be submitted to a journal anytime after the CVPR's submission deadline (even before a final decision on the paper is sent to the authors). An author submitting an extended version of a CVPR paper to a journal needs to ensure that the paper (a) satisfies all submission requirements of the intended journal and (b) does not violate any copyright with IEEE. Authors may also wish to notify the CVPR Program Chairs of their journal submission.

Note that a Technical Report (departmental, arXiv.org, etc.) that is put up without any form of direct peer-review is NOT considered a publication and is therefore allowed, but should NOT be cited. Likewise, mention of the work under review in a presentation is NOT considered a violation. For further information, please refer to Section 8.2.4.F of the IEEE PSPB Operations Manual.

Plagiarism: Plagiarism is a regular nuisance in publication. Plagiarism consists of appropriating the words or results of another, without credit. Generally, reviewers can recognize plagiarism when they see it; it is unlikely that a reviewer will be uncertain whether plagiarism has occurred. CVPR 2019's policy on plagiarism is to refer suspected cases to the IEEE Intellectual Property office, which has an established mechanism for dealing with plagiarism and wide powers of excluding offending authors from future conferences and from IEEE journals. You can find information on this office, their procedures, and their definitions of five levels of plagiarism at this webpage.

© 2019 CVPR