Call for Artifacts

All papers accepted at both EuroSys’24 submission rounds are encouraged to participate in the artifact evaluation process.

Artifacts must be consistent with the paper, as complete as possible, documented reasonably well, and easy to reuse. These ideals are implemented through three badges that can be awarded to each paper: available, functional, and reproduced. The goal of the AEC is to help authors achieve these goals and to award badges to the artifacts that meet the criteria.

Questions about artifact evaluation can be directed to aec-2024@eurosys.org.

For an artifact to be considered for the AEC process, at least one contact author for the submission must be reachable and respond to questions in a timely manner during the evaluation period to allow a sufficient amount of round trips between the AEC and the authors.

Check the dates here

Registration and Submission

Link to HotCRP portal - Spring round: https://eurosys24ae-spring.hotcrp.com/

Link to HotCRP portal - Fall round: https://eurosys24ae-fall.hotcrp.com/

Please submit your artifacts to the AE HotCRP portal and follow the two-step process:

  1. Registration: Register your accepted paper for artifact evaluation by providing the paper’s abstract and PDF.

  2. Submission: Submit your artifact for evaluation by providing an URL or a packaged artifact, selecting which artifact badges you apply for, and provide an artifact appendix that describes the artifact.

The effort that you put into packaging your artifacts has a direct impact on the committee’s ability to make well-informed decisions. Please package your artifacts with care to make it as straightforward and easy as possible for the AEC to understand and evaluate their quality.

Note: If you need permission from your organization’s legal or IT department to publish your artifact or give evaluators access to custom hardware, submit that request as soon as possible, otherwise evaluators may not have sufficient time to audit your artifact.

Process

Authors are invited to submit artifacts right after their papers have been accepted.

Artifacts can be submitted only in the AE time frame associated with the paper submission round (e.g., an artifact for a paper accepted in the Spring submission round cannot be submitted to the EuroSys AE Fall round).

Because the time between paper acceptance and artifact submission is short, the AEC chairs encourage authors to start preparing artifacts while their papers are still under review.

At artifact submission time, authors choose which badges they want to obtain: available, functional, and reproduced. Artifacts can meet the criteria of one, two, or all three of the badges and will be awarded zero to three badges after evaluation.

After the artifact submission deadline, artifact evaluators will evaluate each artifact, using the corresponding paper and artifact appendix as guides. Evaluators may communicate with authors (uniquely through HotCRP to preserve anonymity) to resolve minor issues and ask clarifying questions throughout the whole evaluation process. Evaluation starts with a “kick-the-tires” period during which evaluators ensure they can access their assigned artifacts and perform basic operations such as compiling and running a minimal working example. Artifact evaluations include feedback about the artifact, giving authors the option to improve their artifact and paper using this feedback.

Artifact Details

Artifacts can be software, data sets, survey results, test suites, mechanized proofs, and so on.

Paper proofs are not accepted, as evaluators lack the time and often the expertise to carefully review them. Physical objects, such as computer hardware, cannot be accepted due to the difficulty of making them available to evaluators. To the extent possible, artifacts should be able to run on commodity hardware (i.e., laptop/desktop-like systems). If this requirement cannot be met, please contact the AEC chairs beforehand to accommodate access to special hardware (e.g., your own) for the evaluators.

When submitting your artifact, please specify which of the three badges you are applying for. For the Artifact Functional and Results Reproduced badges, AEC members will attempt to use your artifact to run the experiments described in your paper.

Submitting an artifact for evaluation does not give the AEC permission to make its contents public or to retain any part of it after evaluation. Thus, authors are free to include proprietary models, data files, or code in artifacts. Participating in artifact evaluation does not require the public release of artifacts, though it is highly encouraged.

Detailed artifact packaging instructions are available here.

AEC members may contact authors during the evaluation period, for example to ask for help if they are unable to get the artifact to work, and authors are asked to be responsive to such requests. However, your goal as an author should be to present and document your artifact in a way that AEC members can use it and complete the evaluation successfully in an autonomous manner (ideally without having to interact with the authors). To ensure that your instructions are complete, we suggest that you run through them on a fresh setup prior to submission, following exactly the instructions you have provided.

Review and Anonymity

Artifact evaluation is “single blind”: the identities of artifact authors will be known to evaluators, but authors will not know which evaluators have reviewed their artifacts.

To maintain the anonymity of evaluators, artifact authors should not embed analytics or other tracking tools in the websites for their artifacts for the duration of the artifact evaluation period. If you cannot control this, please do not access this data. This is important to maintain the confidentiality of the evaluators. In cases where tracing is unavoidable, authors should notify the AEC chairs in advance so that AEC members can take adequate safeguards.