PERCOM 2020 will, for the first time, provide the opportunity for authors of accepted papers to participate in an Artifact Evaluation (AE) process. Authors who choose to participate can provide supplementary materials as artifacts (tools, applications, data sets, etc.) that will be reviewed solely for functional completeness (e.g., does provided code compile and run as expected; do experimental results match expected results, etc.). Participating authors will be allotted two additional pages in a separate Artifacts section of the proceedings to describe the nature and contents of the artifact package. Submission of artifacts packages will have no bearing on paper acceptance or publication. However, papers that have verified artifacts or data will be associated with badges in the conference program and proceedings. Two badges will be available: Artifact-Certified and Result-Certified. Upon submission, participating authors should indicate which badge(s) the committee should consider for the submission.
Artifacts of interest include (but are not limited to) the following:
- Software, including implementations or simulations of systems or algorithms.
- Data repositories, including, e.g., logging data, system traces, survey raw data.
- Frameworks, including tools and services illustrating new approaches to pervasive computing systems and application development.
This list is not exhaustive; authors with questions about the suitability of an artifact should reach out to the chairs.
The submission process is straightforward and follows the following steps:
- Package software and datasets. Use a freely available tool, such as VirtualBox; Docker container image; git/mercurial/svn repositories; or zip/tar files.
- Create an artifact guide. The guide can be up to 2 pages and should be prepended to a preliminary version of the accepted paper (as a single PDF file). The guide’s contents will vary based on the artifact but it should describe minimal software and hardware requirements, how to obtain and deploy the artifact, and what a reviewer should expect (e.g., the expected experimental results, or steps to navigate an application). The artifact guide should include a link to a publicly downloadable artifact, whatever that means for your particular submission.
- Submit the artifact guide via EDAS at https://edas.info/newPaper.php?c=26312&track=100881.
The Review Process
The review process will have three phases:
- Initial Review. The reviewers will follow the instructions given in the artifact’s guide to ensure that artifacts can be deployed and run as described. Any bugs or questions will be documented as feedback.
- Artifact Revision. During the second phase, authors will be able to correct any bugs in their artifacts/guides to make sure reviewers can un-package and run the artifact before the deeper evaluation.
- Final evaluation. Reviewers perform their final evaluation to ensure that each artifact runs as expected and/or contains the expected data.
Note that the first two phases may be intertwined (several rounds) if authors and reviewers are quick their communication; every submission is guaranteed at least one opportunity for revision.
Best Artifact Award
A PERCOM 2020 Best Artifact Award will be presented during the conference.
- Registration of intent to submit an artifact: January 15, 2020
- Initial artifact submission deadline: January 22, 2020
- Artifact feedback and revision period: January 22-February 3, 2020
- Artifact evaluation notification (badge allotment): February 5, 2020 [tentative]
- Artifact guide camera ready deadline: February 7, 2020 [tentative]
For further information about the artifact submission and evaluation, please contact the Co-Chairs, Vaskar Raychoudhury (firstname.lastname@example.org), Md Osman Gani (email@example.com), and Christian Becker (firstname.lastname@example.org).