x

We evaluate the submitted artifacts for three distinct badges (selected from amongst the options outlined by the ACM). An artifact can qualify for one, two, or all three of the badges. Please read carefully on the expectations that need to be met for each of these badges. We list them below in the increasing order of rigor:

1. Artifact Available

To earn this badge, all artifacts associated with the paper must be made available for retrieval--permanently and publicly. Valid hosting options include institutional repositories (e.g., the ACM digital library) and open commercial repositories (e.g., github and bitbucket), but not personal webpages. Other than making the artifacts available, this badge does not mandate any further requirements on functionality, correctness, or documentation. Artifacts in this category include both software systems and datasets.

2. Artifact Functional

To earn this badge, all artifacts associated with the paper must undergo an independent audit. While it is not necessary to be publicly released, the artifacts must be made available to AEC members. The submitted artifacts will be evaluated on three asepcts: (i) Documentation: are they sufficiently documented to enable using them independently? (ii) Completeness: do they include all key components described in the paper? (iii) Exercisability: do they include scripts and data needed to run the experiments described in the paper?

3. Results Replicated

To earn this badge, the main results of the paper have to be independently obtained by the evaluating AEC team. While this badge achieves the highest level of scientific rigor, it requires time committment and open communication by both authors and evaluators. All the required artifacts must be provided by the authors. The goal of this effort is not to replicate the results exactly, but instead to generate results independently within the allowed tolerance such that the main claims of the paper are validated.