Submitted artifacts could select to be evaluated against the following badges:
|Artifacts Available: To earn this badge, the AEC must judge that the artifacts associated with the paper have been made available for retrieval, permanently and publicly. Valid hosting options include institutional repositories and third-party digital repositories (e.g., Zenodo, FigShare, Dryad, Software Heritage, GitHub, or GitLab — not personal webpages. Other than making the artifacts available, this badge does not mandate any further requirements on functionality, correctness, or documentation.|
|Artifacts Functional: To earn this badge, the AEC must judge that the artifacts conform to the expectations set by the paper in terms of functionality, usability, and relevance. In short, do the artifacts work and are they useful for producing outcomes associated with the paper? The AEC will consider three aspects of the artifacts in particular.
Documentation: are the artifacts sufficiently documented to enable them to be exercised by readers of the paper?
Completeness: do the submitted artifacts include all of the key components described in the paper?
Exercisability: do the submitted artifacts include the scripts and data needed to run the experiments described in the paper, and can the software be successfully executed?
|Results Reproduced: To earn this badge, the AEC must judge that they can use the submitted artifacts to obtain the main results presented in the paper. In short, is it possible for the AEC to independently repeat the experiments and obtain results that support the claims made by the paper? The goal of this effort is not to reproduce the results exactly, but instead to generate results independently within an allowed tolerance such that the main claims of the paper are validated.|