Report

We are happy to report about the OSDI ‘22 artifact evaluation process. This is the third time that OSDI conducted such a process and we hope to keep improving it so that artifact evaluation will become more common in our community’s conferences. This year, the OSDI ‘22 artifact evaluation process is combined with USENIX ATC ‘22. The combined artifact evaluation committee consists of 118 artifact reviewers from academia and industry.

Process

We continued to use the three-badge approach (vs. the single-badge approach) from OSDI ‘21 evaluation and these three badges include:

• Artifacts Available: To earn this badge, the AEC must judge that the artifacts associated with the paper have been made available for retrieval, permanently and publicly.
• Artifacts Functional: To earn this badge, the AEC must judge that the artifacts conform to the expectations set by the paper in terms of functionality, usability, and relevance.
• Results Reproduced: To earn this badge, the AEC must judge that they can use the submitted artifacts to obtain the main results presented in the paper.

Evaluation

In the evaluation process, each artifact was evaluated by 3 reviewers. The evaluation process had two key phases: the kick-the-tires phase and the in-depth evaluation phase. During the kick-the-tires phase, reviewers made a quick first pass over all assignments to identify and report obvious problems and communicated them with the authors. After the kick-the-tires phase, reviewers evaluated each assignment thoroughly and wrote detailed reviews. Finally, reviewers coordinated and communicated with fellow AEC members and decided which badges should be awarded to each artifact.

Results

OSDI ‘22 accepted 49 papers and 35 papers participated in the AE. Of the 35 submitted artifacts:

• 33 artifacts received the Artifacts Available badge (94%).
• 31 artifacts received the Artifacts Functional badge (88%).
• 27 artifacts received the Results Reproduced badge (77%).

Key Takeaways

Our experience shows that after the kick-the-tires response period, reviewers can still encounter technical problems that obstruct their evaluation. We suggest future AEC chairs extend the kick-the-tires response period and encourage more interaction between reviewers and authors during the period.

Finally, we deeply thank the authors and the AEC committee for all their efforts in making the OSDI ‘22 artifact evaluation possible.

Anuj Kalia, Microsoft
Neeraja J. Yadwadkar, University of Texas at Austin
Chengyu Zhang, ETH Zurich
OSDI ‘22 Artifact Evaluation Committee Co-chairs