IJCAI review form
From Semantic Portal Wiki
IJCAI-09 Reviewing Criteria
All IJCAI-09 papers should be solid scientific papers, regardless of their specific area. Your review should be constructive, thorough, and polite.
We judge the merit of a paper based on six criteria: Relevance; Significance; Technical soundness; Novelty; Quality of Evaluation; and Clarity. For each criterion you have to assign a score between 1 and 10. You should also provide detailed comments justifying your evaluation along with suggestions for improving the paper. Furthermore, please provide specific information on which issues you would like the authors to address in their rebuttal.
Below we provide a detailed explanation of the different criteria and scores. While all criteria are important, we want to encourage highly original and novel papers, and therefore would like you to pay special attention to the "Novelty" criterion.
Here is some guidance on the meaning of the criteria and their numerical scores.
OVERALL SCORE
1. Trivial or wrong or known.
Clearly below IJCAI threshold, I assume no further discussion is needed.
2. A strong rejection.
I will strongly argue for rejection.
3. A clear rejection.
I vote and will argue for rejection.
4. An OK paper, but not good enough. A rejection.
I vote for rejecting it, although would not be upset if it were accepted. It would not be a disservice to the technical program.
5. Marginally below the acceptance threshold.
I lean toward a vote for rejection, but having the paper in the program would not be that bad.
6. Marginally above the acceptance threshold.
I toward a vote for acceptance, but leaving it out of the program would not be a disservice to the technical program.
7. Good paper, accept.
I vote for acceptance. I believe it should be accepted but could understand others having reasons for rejecting the paper.
8. Top 50% of accepted IJCAI papers, a very good paper, a clear accept.
I vote and argue for acceptance.
9. Top 15% of accepted IJCAI papers, an excellent paper, a strong accept.
I will fight for acceptance.
10. Top 5% of accepted IJCAI papers, a seminal paper for the ages.
Clearly an outstanding paper. I assume no further discussion is needed.
CONFIDENCE SCORE
1: The reviewer's evaluation is an educated guess and it is quite likely that the reviewer did not understand central parts of the paper. Either the paper is not in the reviewer's area, or it was extremely difficult to understand.
4: The reviewer is fairly confident that the evaluation is correct. It is possible that the reviewer did not understand certain parts of the paper, or that the reviewer was unfamiliar with some aspect of the relevant literature. Mathematics and other details were not carefully checked.
7: The reviewer is confident but not absolutely certain that the evaluation is correct. It is unlikely but conceivable that the reviewer did not understand certain parts of the paper, or that the reviewer was unfamiliar with a piece of relevant literature.
10: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature.
Relevance What is the relevance of this paper to an AI audience?
1: Not relevant
4: Moderately relevant
7: Relevant to researchers in very specialized subarea of AI
10: Relevant to general AI or a broad subarea of AI
Significance Are the results important? Are other people (practitioners or researchers) likely to use these ideas or build on them? Does the paper address a difficult problem in a better way than previous research? Does it advance the state of the art in a demonstrable way? Does it provide unique data, unique conclusions on existing data, or a unique theoretical or pragmatic approach?
1: Not significant
4: Moderately significant
7: Significant
10: Highly significant
Technical soundness Is the paper technically sound? Are the concepts correct and accurate? Are proofs of results or experimental protocols valid? If theoretical proofs or empirical evidence is lacking for some claims (e.g., due to space limitations), are results technically plausible?
1: Has major errors
4: Has minor errors
7: Technically sound
10: Major technical contribution
Novelty Are the problems or approaches novel? Is this a novel combination of familiar techniques? Is it clear how this work differs from previous contributions? Is related work adequately referenced?
1: Not novel
4: Moderately novel
7: Novel
10: Very novel
Quality of Evaluation Are claims well-supported by theoretical analysis or experimental results? How convincing is the evidence in support of the conclusions? Are the authors careful (and honest) about evaluating both the strengths and weaknesses of the work?
1: Not convincing
4: Moderately convincing
7: Convincing
10: Very convincing
Clarity Is the paper clearly written? Is it well-organized? (If not, feel free to make suggestions to improve the manuscript.) Does it adequately inform the reader? (A superbly written paper provides enough information for the expert reader to reproduce its results.)
1: Poor
4: Satisfactory
7: Good
10: Excellent
Oral/Poster Presentation
IJCAI does not have a separate oral/poster track and no predefined targets for oral/poster presentations. Depending on acceptance rates, all papers may be presented orally, or some will be designated for poster presentation. You are asked to make an assessment as to the suitability of each paper to either oral or poster presentation. Acceptance decisions will be entirely detached from oral/poster decisions. Your recommendations for paper acceptance must be based on the quality of the paper according to the review criteria. Any accepted paper, whether oral or poster, must meet the standards of excellence expected of IJCAI papers. Your oral/poster recommendations will not influence acceptance decisions in any way.
Your recommendation for oral presentation or poster presentation should be based on your assessment of: the overall level of interest in the paper among a broad AI audience; whether the material is well-suited to a less interactive oral presentation or could be best presented to a smaller audience with more personal interaction; the potential for impact of the work/results across AI or a broad AI subdiscipline.
ASIDE: Obviously, oral/poster recommendations will be somewhat *correlated* with your acceptance/rejection recommendations; e.g., papers with broad anticipated impact are more likely to be recommended for acceptance and recommended for oral presentation. However, it is perfectly reasonable to rate paper A more highly than paper B with respect to acceptance, and still recommend that A is best suited for poster presentation and B for oral.
Please choose one of the following:
1. This paper is best suited to oral presentation.
- the paper should be of interest to a broad segment of the AI community - the material is easily explained and presented in oral (non-interactive) mode - this paper is likely to have a very important impact in a key subfield of AI
2. This paper could be presented reasonably well orally or in a poster session.
3. This paper is best suited to poster presentation.
- the paper's results will be of interest to a fairly specialized AI subcommunity; - the material is most easily conveyed through interaction with the audience (e.g., the ability to dwell on details and ask questions is important to understanding key findings in the work) - the results of this paper advance the state of the art in reasonable, but somewhat incremental, fashion
=======Review Form Start===============================
[Please complete this form offline and paste into the "Comments for Author" box in ConfMaster]
Relevance
What is the relevance of this paper to an AI audience?
Significance Are the results important? Are other people (practitioners or researchers) likely to use these ideas or build on them? Does the paper address a difficult problem in a better way than previous research? Does it advance the state of the art in a demonstrable way? Does it provide unique data, unique conclusions on existing data, or a unique theoretical or pragmatic approach?
Technical soundness Is the paper technically sound? Are the concepts correct and accurate? Are proofs of results or experimental protocols valid? If theoretical proofs or empirical evidence is lacking for some claims (e.g., due to space limitations), are results technically plausible?
Novelty Are the problems or approaches novel? Is this a novel combination of familiar techniques? Is it clear how this work differs from previous contributions? Is related work adequately referenced?
Quality of Evaluation Are claims well-supported by theoretical analysis or experimental results? How convincing is the evidence in support of the conclusions? Are the authors careful (and honest) about evaluating both the strengths and weaknesses of the work?
Clarity Is the paper clearly written? Is it well-organized? (If not, feel free to make suggestions to improve the manuscript.) Does it adequately inform the reader? (A superbly written paper provides enough information for the expert reader to reproduce its results.)
[Please answer these questions offline and paste into the "Confidential Comments for PC" box in ConfMaster]
Confidential Comments about paper:
Oral/Poster Presentation (X beside recommendation)
This paper is best suited to oral presentation _____________
This paper could be presented reasonably well orally or in a poster session__________
This paper is best suited to poster presentation_______________
