Important Details

Instructions for Authors

Authors should strive to package their artifacts in a way that makes it easy for the AEC members to use and evaluate them. For most artifacts, this translates to packaging all the required software within a bootable VM or as a Docker image. However, given the breadth of systems covered at SOSP, we will let the authors use their judegement.

1. Determine target badge type(s): The first step is to determine the type(s) of badge the authors intend to seek for their artifact. As noted in the badges section, the authors can apply for multiple badges; the authors can also apply to have their artifacts evaluated without making them public. We encourage the authors to pursue all applicable badges to increase the imapct of the work. Then, register on HotCRP to declare your intent to participate in the AEC process.

2. Package and submit: Next, create a package that contains the following items (i) fully self-contained artifact, (ii) a readme file that describes the contents of the artifact, and setup instructions, (iii) for authors targeting Artifact Functional and Results evaluated, an experiments file that outlines how to run and reproduce the key results of the paper. Upload this package into HotCRP using a compressed archive format like tar-zip. Please prepare the instructions such that a competent systems person can understand and follow them.

3. Communicate and engage with AEC: We expect each artifact to be evaluated by 2 AEC members. While AEC members have strong expertise in systems, the time available for evaluation is limited. Thus, it is important for the authors to interact (anonymously via HotCRP) with their assigned AEC members to provide any clarifications or logistical support towards completing the process in a timely manner.

Instructions for AEC Members

As AEC members, you will familiarize yourself with the research underlying the artifact, and then independently evaluate the submitted artifact against the target badges. We urge you to exercise utmost care and deliberation throughout the process, including but not limited to (i) avoiding accidentally making them public, (ii) limiting all the discussions to within the AEC, and (iii) refraining from posting debugging questions on public forums. Finally, since AEC is single-blind for authors, any communication with them must be done anonymously via HotCRP.

1. Familiarize with the underlying research: The first phase is to acclimatize yourself with the assigned papers, and prepare the groundwork for actual evaluation. You are encouraged to discuss with other AEC members (via the slack channel created for AEC) and seek any clarifications from the authors. Timeline: Aug-3 to Aug-8.

2. Evaluate the artifact(s): The goal is to independently evaluate the submitted artifact against the target badge(s). Start with the author uploaded packages, and follow the instructions outlined in readme and/or experiments files. The effort required for this step would vary considerably based on the complexity of the artifacts as well as target badges. As this is a time-sensitive task, it is best to seek clarifications from your fellow reviewers and the authors, whenever you feel blocked. Timeline: Aug-9 to Aug-28.

3. Assign badges and submit reviews: The last step is to write a detailed review of your evaluation experience and to assign badges. Since artifact submissions are voluntary, and for the benefit of the community, we request you to keep a positive tone in your reviews. Providing precise and descriptive reviews (say, around ~500 words) is likely to help the authors in further improving their artifacts. Timeline: Aug-16 to Aug-28.