1. Artifact Available
To earn this badge, all artifacts associated with the paper must be made available for retrieval--permanently and publicly. Valid hosting options include institutional repositories (e.g., the ACM digital library) and open commercial repositories (e.g., github and bitbucket), but not personal webpages. Other than making the artifacts available, this badge does not mandate any further requirements on functionality, correctness, or documentation. Artifacts in this category include both software systems and datasets.
2. Artifact Functional
To earn this badge, all artifacts associated with the paper must undergo an independent audit. While it is not necessary to be publicly released, the artifacts must be made available to AEC members. The submitted artifacts will be evaluated on three asepcts: (i) Documentation: are they sufficiently documented to enable using them independently? (ii) Completeness: do they include all key components described in the paper? (iii) Exercisability: do they include scripts and data needed to run the experiments described in the paper?
3. Results Replicated
To earn this badge, the main results of the paper have to be independently obtained by the evaluating AEC team. While this badge achieves the highest level of scientific rigor, it requires time committment and open communication by both authors and evaluators. All the required artifacts must be provided by the authors. The goal of this effort is not to replicate the results exactly, but instead to generate results independently within the allowed tolerance such that the main claims of the paper are validated.