Reviews have a direct and important impact on the quality of a conference. They also help the community as a whole to improve the quality of its research. The aim of this guide is to ensure that the review process is fair and leads to the best possible selection of papers. All reviewers should make sure that they follow the basic principles outlined in this text.
The EuroVis reviewing process has a very tight schedule, and an important part of making the commitment to review is that you have agreed to carry out the work before the stated deadlines. All reviewers need to finish reviews on time, and program committee members need to bid and declare conflicts on time.
As a reviewer for EuroVis, you have the responsibility to protect the confidentiality of the ideas represented in the papers you review. Submissions are by their very nature not published documents. The work is considered new or proprietary by the authors; otherwise they would not have submitted it. Of course, their intent is to ultimately publish to the world, but most of the submitted papers will not appear in the proceedings of the conference. Thus, it is likely that the paper you have in your hands will be refined further and submitted to some other journal or conference, or even to the same conference next year. Sometimes the work is still considered confidential by the author's employers. These organizations do not consider sending a paper for review to constitute a public disclosure. Protection of the ideas in the papers you receive means: Do not show the paper to anyone else, including colleagues or students, unless you have asked them to write a review, or to help with your review. Do not show movies or other supplemental material to non-reviewers. Do not use ideas from papers you review to develop new ones. After the review process, destroy all copies of papers and supplemental material that are not returned to the senior reviewer and erase any implementations you have written to evaluate the ideas in the papers, as well as any results of those implementations.
As a reviewer of a EuroVis paper, you have a certain power over the reviewing process. It is important for you to avoid any conflict of interest. Even though you would, of course, act impartially on any paper, there should be absolutely no question about the impartiality of review. Thus, if you are assigned a paper where your review would create a possible conflict of interest, you should return the paper and not submit a review. Conflicts of interest include (but are not limited to) situations in which:
The publishing of scholarly work is essential for the academic community specifically and the research community in general. Therefore careers and reputations hinge on publishing in the proceedings, academic tenure decisions are based on the proceedings, and patent infringement cases discuss whether something is considered novel enough to publish in the proceedings. Therefore, it is our duty to do a careful, objective and scholarly review. The emphasis of our reviews should be to help the authors on how to improve their work and therefore to improve the overall quality of the work in our research community. In general, it will not be helpful to anyone - neither the program chair, nor the authors:
A casual or flippant review of a paper that the author has seriously submitted is not appropriate and may be rescinded from the reviewing process. In the long run, casual reviewing is a most damaging attack on our conference. There is no dishonor in being too busy to do a good review, or to realize that you have over-committed yourself and cannot review all the papers you agreed to review. But it is a big mistake to take on too much, and then not back out early enough to allow recovery. If you cannot do a decent job, give the paper back and say so. But please, do it early so that the senior reviewer has time to select another reviewer before the deadline.
Have an open mind, or at least reveal your biases. If you are a die-hard algorithm-driven researcher, and you are assigned a user study paper, and the call for papers welcomes both types of papers, don't bash the paper simply for its methodology. It's not fair and it benefits no one. Either assess the paper according to the appropriate user-study standards, or admit that you are not capable of doing so. At most, you might discuss why the algorithmic approach does not provide a suitable answer to the research question. But don't force the author to ask a different question that can be answered using algorithmic methods. Many of us have been frustrated by reviews in which the judge basically told the researcher to do a different study. Work within the author's goals.
No matter how much you dislike the paper, try to find the good aspects. Perhaps there is a different approach to a problem, a novel application, a promising evaluation methodology, or even a potential twist which could be studied further during the revision of a paper. Be helpful to the authors and point out the potential positive aspects. Let the authors know if you think their work would be better suited for a different conference, journal or venue. Be encouraging.
Belittling or sarcastic comments may help display one's wit, but they are unnecessary in the reviewing process. The most valuable comments in a review are those that help the authors understand the shortcomings of their work and how they might improve it. If you intensely dislike a paper, give it a low score. That makes a sufficient statement. While we will assure the anonymity of the reviewing process, read over your review and ask yourself whether you would be able to recite your critique in front of the authors? If the answer is 'no' you might want to work on some of the wording of the criticism. A scathing review is usually not only unhelpful to the authors and paper chairs, but it also tends to alienate typically new researchers that would like to enter into our community. Put out a helping hand.
Adherence to ethics sometimes requires a more careful analysis of the work as well as a well- thought out response. However, while it appears that we might be losing some time, this will be made up by improving the culture in our community overall, by helping each other to put out better results and overall increasing the quality of our conferences and journals. That is what we are striving for in our visualization community and it is well worth the effort.