Call for Artifacts

Overview

A scientific paper consists of a constellation of artifacts that extend beyond the document itself, possibly including software, hardware, evaluation data, documentation, survey results, mechanized proofs, models, test suites, and benchmarks. In some cases, the quality of these artifacts is as important as the document itself. FAST ‘24 will run an artifact evaluation process for accepted papers.

The artifact evaluation process will consider the availability and functionality of artifacts associated with their corresponding papers, along with the reproducibility of the paper’s key results and claims with these artifacts. Artifact evaluation is single-blind. Artifacts will be held in confidence by the evaluation committee.

All accepted FAST papers are encouraged to participate in artifact evaluation. Because the time between paper acceptance and artifact submission is short, we strongly encourage authors to start preparing their artifacts for evaluation while their papers are still under consideration by the FAST Program Committee. See the Submitting an Artifact section for details on the submission process.

Artifact evaluation is optional, although we hope all accepted papers will participate. Artifact evaluation will begin only after paper acceptance decisions have already been made.

Questions about the process can be directed to fast24aec@usenix.org.

Important Dates

  • Notification for paper authors: Friday, December 8, 2023
  • Artifact registration deadline: Thursday, December 14, 2023, 8:59 pm PST
  • Artifact submission deadline: Thursday, December 21, 2023, 8:59 pm PST
  • Artifact decisions announced: Monday, January 22, 2024
  • Final paper files due: Monday, January 29, 2024

Note: In the rather common event of questions or discrepancies in using or understanding the artifacts, at least one contact author for the submission must be reachable via email and respond to questions in a timely manner during the artifact evaluation period.

Benefits and Goals

The dissemination of artifacts benefits science and engineering. Their availability encourages reproducibility and enables authors to build on top of each others’ work. It can also unambiguously resolve questions about cases not addressed by the original authors. It also confers direct and indirect benefits to the authors themselves, such as a fresh set of eyes checking the experimental setup steps or confirming someone other than the authors can use the released code.

The goal of artifact evaluation is to incentivize authors to invest in their broader scientific community by producing artifacts that illustrate their claims, enable others to validate those claims and accelerate future scientific progress by providing a platform for others to start from. A paper with artifacts that have passed the artifact evaluation process is recognized in two ways: first by badges that appear on the paper’s first page, and second by an appendix that details the artifacts.

Criteria

Each paper sets up certain expectations and claims of its artifacts based on its content. The AEC will read the paper and judge whether the artifacts match those criteria. Thus, the AEC will decide that the artifacts do or do not “conform to the expectations set by the paper.” Ultimately, the AEC expects that high-quality artifacts will be:

  • consistent with the paper
  • as complete as possible
  • documented well
  • easy to reuse, facilitating further research

Acknowledgements

FAST ‘24 AE was inspired by multiple other conferences, such as OSDI, USENIX Security, SOSP, and several SIGPLAN conferences. See artifact-eval.org for the origins of the AE process and sysartifacts.github.io for the previous AE processes held in systems.