Evaluation

Scientific papers consist of artifacts that extend beyond documents: software, hardware, data and documentation, raw survey results, mechanized proofs, models, test suites, benchmarks, and so on. The quality of these artifacts is as important as that of the document itself.

The dissemination of artifacts benefits science as a whole:

• It encourages replication and reproduction, enabling scientists to extend previous work
• It helps resolve questions about cases not considered by the original authors

Polishing and publishing an artifact also benefits the authors:

• Scientific work has much more impact (and citations…) if it is documented and polished enough that others can build on it and use it as a comparison point
• It is much easier to onboard new collaborators if tools are documented and their setup is automated

Artifact evaluation is an incentive for authors to invest time to produce quality artifacts. A paper whose artifacts have passed the artifact evaluation process includes an appendix that details the artifacts and is recognized by one or more badges that appear on the paper’s first page.

Based on the success of artifact evaluation in previous conferences, EuroSys will continue to run an optional artifact evaluation process. Eventually, the assessment of a paper’s accompanying artifacts may guide the decision-making about papers: that is, the Artifact Evaluation Committee (AEC) would inform and advise the Program Committee (PC). At EuroSys 2022, artifact evaluation will begin only after paper acceptance decisions have already been made.