Call for Artifacts
Artifact Submission Closed.
CCGrid 2023 will award the following two IEEE reproducibility badges to papers that have been accepted to the main Research Tracks: Open Research Objects (ORO) and Reusable/Research Objects Reviewed (ROR). Authors may optionally request the badge(s) for their accepted paper by submitting relevant artifact(s) and a 2-page artifact description, and be awarded the badge(s) if their submission is successfully reviewed. The badges will appear as part of the paper in the conference proceedings.
All deadlines are in Anywhere on Earth (AoE), UTC-12 timezone.
- Artifact Submission for Accepted Papers: Feb 5th February 10th, 2023
- Artifact Review Midpoint Check-in (reviewers contact authors initially if needed): February 18th, 2023
- Artifact Review Notification: March 8th, 2023
- Daniel S. Katz, University of Illinois at Urbana-Champaign (UIUC), USA (email@example.com)
- Diana Moise, Cray (HPE), Switzerland (firstname.lastname@example.org)
Open Research Objects (ORO)
This badge signals that author-created digital objects used in the research (including data and code) are permanently archived in a public repository that assigns a global identifier and guarantees persistence, and are made available via standard open licenses that maximize artifact availability.
- This is akin to author-supplied supplemental materials, shared under a standard public license such as an Open Science Initiative (OSI)-approved license for software or a Creative Commons license or public domain dedication for data and other materials.
- This definition corresponds to the Association for Computing Machinery (ACM) “Artifacts Available” badge, and to the combined Center for Open Science (COS) “Open Data” and “Open Materials” (pertaining to digital objects) badges.
- The determination of what objects are “relevant” to a research publication is in the hands of the artifact evaluation committee, in addition to the authors themselves.
- For software artifacts, there should be sufficient documentation for the reviewer to understand the core functionality of the software under review, while for data artifacts, there must be adequate documentation, including context, description, how it was sourced, and how it is used in the paper.
- For physical objects relevant to the research, the metadata about the object should be made available.
- Review of both software and data will include the following criteria:
- Availability: Is the data/software permanently archived in a public repository that assigns a global identifier and guarantees persistence
- License: Is the data/software made available via standard open licenses that maximize artifact availability.
- Authorship: Does the authorship associated with the archived artifact appear reasonable and complete?
Reusable/Research Objects Reviewed (ROR)
This badge signals that all relevant author-created digital objects used in the research (including data and software) were reviewed according to the following criteria.
- A publication that is awarded the ROR badge must also be awarded the ORO badge.
- This badge corresponds to the ACM “Artifacts Evaluated” badge, as well as the previously-used IEEE “Code Reviewed” badge.
- Review of a software artifact will include the following criteria, which must be satisfied in the artifact:
- Documentation: There should be sufficient documentation for the reviewer to understand the core functionality of the software under review, including a statement of need/function, installation instructions, examples usage, and API documentation
- Functionality: Reviewers are expected to install the software they are reviewing and to verify the core functionality of the software.
- Testing: Minimally, that the software includes documented manual steps that can be followed to objectively check the expected functionality of the software (e.g., a sample input file to assert behavior); and ideally, that it includes an automated test suite hooked up to continuous integration (GitHub Actions, Circle CI, or similar)
- Review of a data artifact will include the following criteria, which must be satisfied in the artifact:
- Documentation: There must be adequate documentation on any dataset used in the paper, including context, description, how it was sourced, and how it is used in the paper.
- Functionality: The documentation should be sufficient to promote reusability of the data beyond just the current paper. In case of proprietary data formats, there should be associated code to programmatically access the data elements.
We suggest permanent repositories for archival that promote findable, accessible, interoperable and reusable (FAIR) principles such as Zenodo, Dryad or figshare to deposit your artifacts. If you are hosting your software and/or data on GitHub, you should additionally assign it a persistent identifier by linking it to Zenodo, figshare, etc.
Artifact Preparation and Submission Process
Authors seeking one or both of these badges should submit a 2-page artifact description document that includes a brief description of the artifacts and any needed details for them to be reviewed as part of the CCGrid 2023 artifact review process and then used by future readers. For software: a link to the artifact, and a description that includes language, compilation and run environment (tools, pre-built binaries, hardware), input dataset (if available), expected output and metrics, estimated time of all the compilation and run steps, etc. For data: a link to the artifact, and a description as mentioned in the review criteria above. Clearly make connections between the specific artifact(s) and their role and context within relevant parts of the accepted paper. You must also explicitly reference and cite your artifacts in this document, including a persistent identifier to it (e.g., DOI from Zenodo, figshare) and for software, optionally a link to a URL where it is being developed and maintained (e.g., GitHub). Given the 2-page limit, you should include key details in the description document and more exhaustive steps in the persistent artifact link. You should format this document using the IEEE 2-column conference proceedings template. If artifacts are successfully evaluated, the authors will be allowed to add an updated 2-page version of their artifact description as an Appendix to the camera-ready paper. The review of the artifacts will follow a single-blinded process.
The artifact badging process will occur between author notifications (expected 27 January 2023) and the Camera-Ready paper submission deadline (17 March 2023)
Note: Artifacts should be able to run on commodity workstations/laptops for the evaluation. In case the artifact is tightly coupled to a specific class of system or requires more than a generic workstation to be meaningfully evaluated (e.g., an HPC cluster, cloud resources, specialized accelerators, etc.), authors should provide access to such an evaluation environment that can be used by the artifact reviewers. This pre-requisite should be made clear in the Artifact submission and in the EasyChair abstract. The relevant credentials to the specialized resource may be shared by email with the Artifact Evaluation Committee Chairs to be passed onto the reviewers anonymously. If you require further guidance on this, please contact the Artifact Evaluation Committee Chairs ahead of the submission deadline.
We, as the CCGrid community, are committed, both individually and collectively, to nurturing diversity and inclusion, sustaining a climate of mutual respect, and fostering a collaborative and supportive environment wherein everyone is empowered to succeed.
Authors and participants are encouraged to read the ACM Code of Ethics and Professional Conduct and the ACM Policy Against Harassment, and IEEE Computer Society's expectation of professional behavior found in the IEEE Computer Society Open Conference Statement, the IEEE Event Conduct and Safety Statement, and the IEEE Code of Ethics.