Call for Artifacts

All accepted and revision papers are encouraged to participate in artifact evaluation.

Each paper sets up certain expectations and claims of its artifacts based on its content.
This year the Technical Program Committee (TPC) will provide a list of major claims for each accepted paper to the Artifact Evaluation Committee (AEC). The AEC will use this list during the artifact evaluation to ensure that the major claims the TPC deemed important correspond to what the authors indicate and can be reproduced.

Artifacts must be consistent with the paper, as complete as possible, documented well, and easy to reuse. These ideals are implemented through three badges that can be awarded to each paper: available, functional, and reproduced. The goal of the AEC is to help authors achieve these goals, and to award badges to the artifacts that meet the criteria.

Questions about artifact evaluation can be directed to aec-2022@eurosys.org.

Important Dates

All AEC deadlines are Anywhere on Earth (AoE)

  • Acceptance notification to paper authors: January 20, 2022
  • Artifact intent registration deadline: January 26, 2022
  • Artifact submission deadline: February 1st, 2022
  • Kick-the-tires response period: February 2nd - 11th, 2022
  • Artifact decisions announced: March 19, 2022
  • EuroSys final papers deadline: March 19, 2022

Note: For an artifact to be considered, at least one contact author for the submission must be reachable and respond to questions in a timely manner during the evaluation period.

Registration and Submission

Please submit your artifacts to HotCRP and follow the two-step process:

  1. Registration: Register your accepted paper for artifact evaluation by providing the paper’s abstract and PDF.

  2. Submission: Submit your artifact for evaluation by providing an URL or a packaged artifact, selecting which artifact badges you apply for, and, new this year, provide an artifact appendix that describes the artifact and links it to the paper’s claims; see the instructions page for details.

The effort that you put into packaging your artifacts has a direct impact on the committee’s ability to make well-informed decisions. Please package your artifacts with care to make it as straightforward and easy as possible for the AEC to understand and evaluate their quality.

Note: If you need permission from your org’s legal or IT department to publish your artifact or give evaluators access to custom hardware, submit that request as soon as possible, otherwise evaluators may not have enough time to audit your artifact.

Process

Authors are invited to submit artifacts after their papers have been accepted for publication or revision. Because the time between paper acceptance and artifact submission is short, the AEC chairs encourage authors to start preparing artifacts while their papers are still under review.

At artifact submission time, authors choose which badges they want to obtain: available, functional, and reproduced. Artifacts can meet the criteria of one, two, or all three of the badges.

After the artifact submission deadline, artifact evaluators will evaluate each artifact, using the corresponding paper and artifact appendix as guides. Evaluators may communicate with authors (through HotCRP to maintain anonymity) to resolve minor issues and ask clarifying questions. Evaluation starts with a “kick-the-tires” period during which evaluators ensure they can access their assigned artifacts and perform basic operations such as compiling and running a minimal working example. Artifact evaluations include feedback about the artifact, giving authors the option to improve their artifact and paper using this feedback.

Artifact Details

Artifacts can be software, data sets, survey results, test suites, mechanized proofs, and so on. Paper proofs are not accepted, as evaluators lack the time and often the expertise to carefully review them. Physical objects, such as computer hardware, cannot be accepted due to the difficulty of making them available to evaluators. To the extent possible, artifacts should be able to run on commonly-available hardware, or hardware in community research testbeds such as Emulab, CloudLab, and Chameleon Cloud.

Submitting an artifact for evaluation does not give the AEC permission to make its contents public or to retain any part of it after evaluation. Thus, authors are free to include proprietary models, data files, or code in artifacts. Participating in artifact evaluation does not require the public release of artifacts, though it is highly encouraged.

Please see the submission instructions page for further details.

Review and Anonymity

Artifact evaluation is “single blind”: the identities of artifact authors will be known to evaluators, but authors will not know which evaluators have reviewed their artifacts.

To maintain the anonymity of evaluators, artifact authors should not embed analytics or other tracking tools in the websites for their artifacts for the duration of the artifact evaluation period. If you cannot control this, please do not access this data. This is important to maintain the confidentiality of the evaluators. In cases where tracing is unavoidable, authors should notify the AEC chairs in advance so that AEC members can take adequate safeguards.