Status of this document

The instructions below are the reviewing guidelines provided by the PC chairs to the members of the PC and to the reviewers. We make them public in the name of transparency, and in case this helps authors for future editions.

Reviewing and selecting the proposals in light of the goals of our conference can be challenging. The guidelines are meant to flesh out some aspects of the call for presentations, and to ensure that the conference remains varied and on-topic whilst avoiding being artificially selective.

— The organisers

Reviewing guidelines

24th October 2025

Goal

We aim to establish a shortlist of submissions deemed equally deserving of being presented at the conference. The goal is not to establish a ranking that compares submissions (as the contributions are not meant to be comparable). We also do not want to punish submissions for minor shortcomings (i.e. be artificially elitist or reject submissions that do not conform to appearance standards of our research communities).

Key points

  • As per the call for presentations, the contributions are evaluated accoring to three criteria: “computer science”, “author expertise”, and “undone science”. We expand on these criteria further below.
  • As promised in the call for presentations, “we will endeavour to always give considerate and constructive feedback about proposed abstracts”. This is very important for a conference that encourages scientists to get outside of their comfort zones. One way to be constructive is to provide relevant bibliography.
  • Any PC member can spontaneously submit reviews, and participate in discussions of any paper (barring conflicts of interest).
  • Given the quantity and quality of submissions, please be rigorous in the evaluation (without penalizing exploratory work).

Sub-reviewers

PC members have the option of delegating some of their assigned reviews to colleagues outside of the program committee. However, in the case of a double-blind submission, we ask PC members to be mindful of the possibility of conflicts of interest. Subreviewers can for instance be recruited from your own teams/institutions if you think a conflict with the paper is unlikely, but please ask the PC chairs first if you want to recruit an external subreviewer or if you suspect a conflict of interest.

Reminder of acceptance criteria and comments

The following is taken from the call for presentations (additional comments are in italics).

  1. “Computer Science: we seek contributions pertaining to computer science (in a broad sense comprising both works in computer science and works on computer science).”

    • For instance, a paper on a technology’s impact on society would be out of scope (unless it deals with how it is related to the role or the practice of computer science as a scientific field)

  2. “Author expertise: we expect authors to contribute in accordance with their domains of expertise, in a broad sense; for instance a contribution on ethical issues by a computer scientist can be rooted in their research practice, a contribution by a social scientist can be rooted in the study of an example or through field work; a contribution by a philosopher or historian of science can be rooted in the study of computer science as a specific domain of production of knowledge and scientific practice.”

    • The submission can be self-sufficient, but if the author’s background is indicated, it should be relevant and appropriate, especially if the submission is based on their experience as a practitioner.
    • Please take the time to appreciate the quality of the bibliographic work: a talk proposal that appears to ignore significant relevant references should be rated lower.

  3. “Undone science: we expect that the question of undone science will inspire presentations that lead to meaningful reflections touching upon ethical or epistemological aspects of computer science in a broad sense, without requiring expertise in epistemology and ethics. For instance, a submission from a computer scientist could outline a potential ethical question, rooted in their research practice, even if its consequences in ethics cannot be fully elaborated at this point. (In doubt, feel free to inquire with the organisers about a potential topic.)”

    • The undone science aspect needs to be present, but does not need to be explicitly discussed. What is or is not undone computer science is open to interpretation—in fact we have chosen a broader meaning than in the original Hess et al. papers.
    • PC members can choose to reject a contribution without sufficient links to epistemological or ethical questions as being out of scope. But this is not a strict criterion either, in case a really good talk proposal appears to fit undone science, yet is without obvious epistemological or ethical undertones.

  4. “Unfinished or exploratory contributions, that would benefit from discussion at the conference prior to their possible development into full papers, are welcome. There is no submission category, but authors can make it clear in the submission if they strongly prefer a short or a long presentation slot.”

    • Please do not penalize unfinished or exploratory contributions.
    • Submissions can be allotted shorter time slots by the PC. Please indicate in your review if you think the talk would be best as a short talk instead of a full-length one.

Reminder of other relevant instructions for authors

  • “Abstracts should be 1-3 pages in length (excluding bibliography) and should succinctly present the key arguments and contributions of the proposed talk. The submission can contain appendices or a link to a longer version, but the point of the submission should be clear from the first three pages (reviewers are not obligated to read any further).”
  • “Authors can opt for lightweight double-blind reviewing (with the identity of the authors revealed to the reviewers after they submit their review). The biography can still be included in an anonymised form (omitting the names of both people and institutions).”
  • “Recognising the importance of the promotion of multilingualism in science, we welcome submissions in multiple languages, provided that the submission can be reviewed from the English version alone."

About blind reviewing and reviewer biases

We have opted for an optional lightweight double-blind reviewing (defaulting to single-blind otherwise); in fact, we have encouraged authors to include a short bio (possibly with key elements anonymised). Due to the nature of our conference, we believe that the biography can provide useful context for reviewers, especially in the benefit of outsiders. We should be mindful, however, of the risks and biases of single-blind reviewing (regarding, notably, gender but also other factors, e.g. whether the author is famous or the host institution is prestigious). For further reading, see for instance:

  • Kaatz A, Gutierrez B, Carnes M. Threats to objectivity in peer review: the case of gender. Trends in pharmacological sciences. 2014;35(8):371-373.
  • Tomkins A, Zhang M, Heavlin W. Single vs. Double Blind Reviewing at WSDM, https://arxiv.org/pdf/1702.00502.pdf
  • Okike K, Hug KT, Kocher MS, Leopold SS. Single-blind vs Double-blind Peer Review in the Setting of Author Prestige. JAMA. 2016 Sep 27;316(12):1315-6.