Artifact Evaluation

A scientific paper consists of a constellation of artifacts that extend beyond the document itself: software, hardware, evaluation data and documentation, raw survey results, mechanized proofs, models, test suites, benchmarks, and so on. Based on the success of artifact evaluation for past editions, SOSP 2025 will have an artifact evaluation process.

The artifact evaluation process considers the reusability and reproducibility of artifacts associated with papers. Artifact evaluation is single blind: authors do not know evaluators’ identity, but evaluators know who the authors are.

Call for evaluators: Apply here to join the artifact evaluation committee!

Artifact registration and submission: https://sosp25ae.hotcrp.com/

Process

Artifact evaluation is a cooperative process, unlike paper reviewing, meaning that authors have the opportunity to fix problems found during evaluation if they can do so in reasonable time.

  1. Authors decide which badges and associated checklists they target.
  2. Authors register their artifact indicating which badges they target, along with the paper PDF and some metadata.
  3. Authors then submit their artifact, packaged according to the artifact packaging guide using a service such as GitHub.
  4. Evaluators begin with a short “kick the tires” phase to double-check that the artifacts are complete and appear usable and contact authors if that is not the case.
  5. Evaluators proceed with the main evaluation period, operating mostly independently but occasionally contacting authors if an issue arises.
  6. Authors upload the final version of their artifact to permanent storage such as Zenodo and submit their artifact DOI.
  7. Authors are awarded badges which they add to their camera-ready paper submission.

Important Dates

  • Notification for paper authors: July 15, 2025
  • Artifact intent registration deadline: August 1, 2025
  • Artifact submission deadline: August 1, 2025
  • Kick-the-tires response period: August 2 – August 15, 2025
  • Artifact decisions announced: September 1, 2025
  • SOSP final papers deadline: TBA

At least one author for each artifact submission must be reachable via email and respond to questions in a timely manner during the kick-the-tires period. Ideally, authors should also be available to respond to reasonable requests during the main evaluation period.

Awards

Artifacts that obtain all available badges are eligible to receive a Distinguished Artifact award. This award will be given to the artifacts that best exemplify reusability and reproducibility practices, including clear documentation, simple installation, ease of use, and extensibility.

The chairs may, at their discretion, award Distinguished Evaluator awards to evaluators who are particularly effective, friendly, and efficient.

Committee

  • Sanidhya Kashyap, EPFL
  • Sudarsun Kannan, Rutgers University
  • Baptiste Lepers, Inria

Full membership list here.