Call for Artifacts

Overview

A scientific paper consists of a constellation of artifacts that extend beyond the document itself, possibly including software, hardware, evaluation data, documentation, survey results, mechanized proofs, models, test suites, and benchmarks. In some cases, the quality of these artifacts is as important as the document itself. FAST ‘25 will continue the artifact evaluation process established in FAST ‘24 for all accepted papers.

The artifact evaluation process will consider the availability and functionality of artifacts associated with their corresponding papers, along with the reproducibility of the paper’s key results and claims with these artifacts. Artifact evaluation is single-blind. Artifacts will be held in confidence by the evaluation committee. We encourage all the authors to openly publish their artifacts to benefit the broader community.

All accepted FAST papers are encouraged to participate in artifact evaluation. Because the time between paper acceptance and artifact submission is short, we strongly encourage authors to start preparing their artifacts for evaluation while their papers are still under consideration by the FAST Program Committee. See the Submitting an Artifact section for details on the submission process.

Participation in artifact evaluation is optional, although we hope all accepted papers will take part. Artifact evaluation will begin only after paper acceptance decisions have already been made. The decision of artifact evaluation does not affect the paper acceptance decision, however, we encourage authors to make the best efforts to obtain the “Results Reproduced” badge.

Questions about the process can be directed to fast25aec@usenix.org.

Important Dates

  • Notification for paper authors: Friday, December 6, 2024
  • Artifact submission deadline: Thursday, December 19, 2024, 8:59 pm PST
  • Artifact decisions announced: Tuesday, January 21, 2025
  • Final paper files due: Tuesday, January 28, 2025

Note: In the rather common event of questions or discrepancies in using or understanding the artifacts, at least one contact author for the submission must be reachable via email and respond to questions in a timely manner during the artifact evaluation period.

Benefits and Goals

The dissemination of artifacts benefits science and engineering. Their availability encourages reproducibility and enables authors to build on top of each others’ work. It can also unambiguously resolve questions about cases not addressed by the original authors. It also confers direct and indirect benefits to the authors themselves, such as a fresh set of eyes checking the experimental setup steps or confirming that someone other than the authors can use the released code.

The goal of artifact evaluation is to incentivize authors to invest in their broader scientific community by producing artifacts that illustrate their claims, enable others to validate those claims, and accelerate future scientific progress by providing a platform for others to start from. A paper with artifacts that pass the artifact evaluation process is recognized in two ways: first, with badges displayed on the USENIX website and the paper’s first page, and second, with an appendix detailing the artifacts.

Criteria

Each paper sets up certain expectations and claims of its artifacts based on its content. The artifact evaluation committee (AEC) will read the paper and judge whether the artifacts match those criteria. Thus, the AEC will decide whether the artifacts do or do not “conform to the expectations set by the paper.” Ultimately, the AEC expects that high-quality artifacts will be:

  • consistent with the paper
  • as complete as possible
  • documented well
  • easy to reuse, facilitating further research

Acknowledgements

FAST ‘25 AE was inspired by multiple other conferences, such as OSDI, USENIX Security, SOSP, and several SIGPLAN conferences. See artifact-eval.org for the origins of the AE process and sysartifacts.github.io for the previous AE processes held in systems.