Artifact Submission Instructions

Important Dates

MobiSys' 23 Artifact Evaluation

At MobiSys, we aim to promote high-quality research in mobile systems and to advance the state of the art. However, traditional peer-review of papers alone cannot guarantee the reproducibility of the research results. The Artifact Evaluation process complements the paper review process by verifying that the artifacts (e.g., code, data, scripts, or instructions) associated with a paper are correct, complete, and properly documented. The Artifact Evaluation committee also provides feedback to authors to improve the artifacts and ensure that they are reusable by others in the community.

Process Overview

Submission: Authors of conditionally accepted papers should use https://mobisys23artifact.hotcrp.com to submit their conditionally accepted papers for artifact evaluation. We provide guidelines below to prepare research artifacts for submission.

Feedback and Revision: The Artifact Evaluation committee will independently review the submitted artifacts, focusing on reproducibility, functionality, and adherence to the paper's claims. The committee may communicate with the authors during this phase to clarify any questions or request additional information.

Badges and Recognition: The Artifact Evaluation committee will provide feedback to the authors. If required, authors may be asked to revise their artifacts based on the committee's suggestions. The process may involve several iterations to ensure that the artifacts meet the evaluation criteria.

Search and Discoverability: Authors will apply for specific ACM badges for “Artifact available”, “Artifact evaluated - functional”, “Results reproduced” for Artifact Evaluation committee evaluation. These badges are independent, and authors may seek one, two, or all three of the badges for their papers. Badges that pass Artifact Evaluation committee approval will be printed on the papers and included as meta information in the ACM Digital Library. Authors can also include up to two pages of their Artifact Appendix in the camera-ready version of the paper.

Guidelines for the preparation of research artifacts (for the authors)

Artifact Evaluation Methods and Criteria (including badge-specific guidelines)

It is recommended that the artifact evaluation committee members bid on the submitted artifacts, so that the artifacts can be reviewed by reviewers with the appropriate expertise. Conflict of Interests should be handled in the similar way as it is in regular paper submission. The reviewers are encouraged to use the following guidelines as the reference in artifact evaluation, and to communicate with the authors as necessary via the submission site (e.g., HotCRP). At the end of the review period, each reviewer should submit recommendations about the badges to be granted, and it will be the artifact evaluation committee chairs’ discretion to make final decisions.

Artifacts Available

Default ACM guidelines will apply to evaluating this badge. For software artifacts, a publicly accessible DOI or link to the source code repository along with a unique identifier must be provided, in order to receive this badge. For hardware artifacts, similar accessibility to all the source files of hardware designs and source codes of associated firmware should be provided.

Artifacts Evaluated - Functional

For software artifacts that do not involve modification of hardware, we encourage reviewers who own the listed hardware and software systems that are required by the artifacts to bid these artifacts. The reviewers will then use the authors’ submitted workflow scripts to operate the artifacts on the reviewers’ systems, and follow standard ACM guidelines to evaluate for badges of “Artifacts Evaluated – Functional” and “Artifacts Evaluated – Reusable”. During the review period, the reviewers can reach out to authors to ask for further explanations on using the scripts, or request for further revisions on the scripts.

In cases where reviewers with the required hardware or software systems to operate the artifacts cannot be found, remote access to the authors’ machines should be used instead for artifact evaluation. Refer to the following paragraph for remote access.

For hardware artifacts, the authors are responsible to provide remote access for the reviewers to access the authors’ machines and operate the artifacts. Commonly used remote access tools such as TeamViewer can be used for such purpose, and the authors are responsible to schedule evaluation appointments with the reviewers. In evaluation via remote access, the authors should make sure that reviewers have full control over the artifacts and operate the entire artifact workflow, including but not limited to, compiling the source codes, uploading the compiled binaries to the target system, configure the target hardware system, start hardware system operations and execute experiments listed in the artifact appendix. During this procedure, the authors are allowed to communicate with reviewers via audio/video chats and provide necessary instructions and explanations.

Results Reproduced

Reviewers are requested to make judgments based on the key results to be reproduced that are listed in the authors’ artifact appendix being submitted, and evaluate if these key results can be reproduced using the artifact submitted by the authors. It is up to the reviewers’ discretion to decide whether the key results listed by the authors cover all the main results reported in the paper, and reviewers have the right to add more results to be evaluated from those reported in the paper.

However, due to the practical difficulty of evaluating the reproducibility of the following types of artifacts, special evaluation criteria can be applied as described below.

Experiment results that require large amounts of repeating experiments: Given that many key results in mobile systems papers are statistical and hence require large amounts of repeating experiments in the same or different settings, it would be hard to reproduce the exact same experiment results in artifact evaluation. In these cases, upon the mutual agreement between the reviewers and the authors, the authors could prepare and submit a demo video that is made by operating their workflow scripts being submitted and demonstrates all the key system functionalities in the artifact. The reviewers could then follow the workflow scripts and demo video to evaluate if the demonstrated system functionalities can be reproduced/replicated. It is the reviewer’s discretion to decide if the demo video sufficiently reflects the key results being reported in the paper.

For experiment results that are subject to specific environmental settings or conditions:> Similar as above, the authors are responsible to make sure that 1) these environmental settings and conditions are fully reflected in the demo video; and 2) when artifacts are evaluated via remote access, these settings and conditions are fully replicated at the authors’ side. It is the reviewer’s discretion to decide if the environmental settings and conditions match those described in the paper.

For experiment results that involve large amounts of computations, data collections or transmissions that are time consuming: The reviewers and the authors could communicate to decide a reduced but still representative scope of experiments that can be completed within a reasonable time frame. Examples of such reduced but representative scope include a) a subset of a public dataset with the same dimensionality (e.g., number of classes); b) a reduced amount of data transmissions under the same transmission link conditions.

For experiment results that involve human subjects: The authors are responsible to provide access to the system artifacts that are deployed on human subjects with the same conditions and system configurations as described in the paper. Similar to above, the authors are also responsible to provide a demo video that instructs the reviewers to safely operate the artifact over human subjects. Such access, if being provided by remote access, is subject to IRB approval at the authors’ institutions.

Additional information

Disclaimer

This text is an adapted version of the ASPLOS 2022 call for artifacts. We thank ASPLOS 2022 artifact evaluation chairs for this inspiration and Wei Gao (University of Pittsburg), Nirupam Roy (UMD College Park) and Inseok Hwang (POSTECH) for feedback and support.