Context, credits and references
Version of 3 March 2025; CC BY-SA 4.0.
Part of the TIER2 project
The Handbook is part of the TIER2 project, enhancing Trust, Integrity and Efficiency in Research through next-level Reproducibility. TIER2 receives funding from the European Union's Horizon Europe research and innovation programme under grant agreement No 101094817.
The Handbook is one of the eight Pilots in TIER2, designed to develop, implement, and evaluate new reproducibility-related tools and practices, emphasising stakeholder engagement and collaboration throughout the project's duration. One of the two pilots targeted to publishers, the Handbook is known in TIER2 as Pilot 8, and it is described here.
A TIER2 workshop with publishers in May 2023 highlighted the ongoing challenge with the sharing of datasets, code, material and other digital objects underpinning results in publications. A consensus priority among the workshop attendees was the need for a common educational and practical set of checks to help in-house editorial staff managing the manuscripts to ensure digital objects were shared in a FAIR-compliant manner and ultimately reproducibile. However, a key requirement was to have a small number of checks, feasible and achievable. An incremental alignment and consistency across journals and publishers was the message, over an attempt to make a comprehensive but unrealistic list of checks, which ultimately will not be put in practice.
In 2024, the academics and professionals in publishing organisations, listed below, worked to co-define and co-create the Handbook via a series of interactive workshops that focused on two workstreams:
The Checklist workstream: identifications of the checks, definition, value and implementation; where relevant, using FAIRsharing to signpost standards and data resources, and indicate which checks can be automatised and how.
The Workflow workstream: definition of a consensus workflow indicating which roles should perform which checks and when during the internal manuscript process.
Although the primary target audience are in-house editors managing the manuscripts, the Handbook also provide: (i) advice to reviewers, on what compliance to the journal data policy may require, (ii) information for authors on what is expected from them by a number of journals, and (iii) requirements source for developers to drive their service provisions to publishers.
As of February 2025, the intervention is in progress and will run for up to six months, and the results will be published in later this year, along with all underlying data.
Leaders and participants
For more information, please, contact the pilot co-leads.
University of Oxford
University Academic Lead of Research Practice; Prof of Data Readiness; OeRC Director
Co-lead (funded partner)
University of Oxford
FAIRsharing Content and Community Coordinator
Co-lead (funded partner)
Taylor & Francis
Head of Data Initiatives
Co-lead (associate partner)
University of Oxford
DPhil Student: improving research culture and practice
Intervention lead (funded student)
Annie Hill
American Psychological Association
Editorial Director, Journals
Pilot member
Elisa De Ranieri
Cell Press
Head of Research Integrity and Author Experience
Pilot member
Grace Ranola
Journal of Petrology / Oxford University Press
Publisher
Pilot member
Libby Dunkley
Cambridge University Press
Editorial Operations Manager
Pilot member
Louie Sandys
Research Directions, Cambridge University Press
Publisher
Pilot member
Research Directions, Cambridge University Press
Publisher and Programme Manager
Pilot member
Pierre Nauleau
Lancet
Senior Editor
Pilot member
Existing material revised to identify the checks
A number of documents were reviewed in order to create a consensus Checklist that would work for a variety of publishers and that incorporates the most frequently recommended elements of each source.
Those that were used as sources for the checklist are included in the section below, and referenced within the Guidance documents via the labels from the 'Acronym used' column.nThose that were not used as sources for the Checklist (as they were deemed out of scope) are included in a separated section below.
Included Sources
F1000 checks summary
Subject Agnostic
F1000
Software/code
FAIR4RS
F1, F2-F4, R1.1, R3
Excluded as too granular: F1.1 (different granularity getting additional PIDs), F1.2 (different versions get different PIDs), all A principles (dealing with longer-term preservation and resolution of identifiers), all I principles (do journals really check that software interoperates using standard formats?), R1.2 (provenance - out of scope?) and R2 (references to other software)
Life Sciences
GigaScience
Resource subsection; Availability of data and materials subsection
Generally out of scope for this handbook, but some recommendations are at a high-enough level to be included
Life Sciences
MDAR
Specifically, the MDAR checklist for authors. This is also explicitly intended for editors
MDAR checklist items that we are not currently including, as are trying to keep a narrow focus on reusability and reproducibility: study-level protocols, study design, statistical tests, attrition. See also MDAR here: https://www.ncbi.nlm.nih.gov/books/NBK556610/, incl comments about how 'less is more' in terms of checklist items, to increase completeness. This can be partially accomplished through domain-specific formats being used for the digital objects in the first place, and the completeness of the metadata in those formats. That lessens the burden for the editors/reviewers.
Subject Agnostic
Nature
Code and data availability
Life Sciences
NIH
Note that the NIH states that "Journals should use a checklist during editorial processing to ensure the reporting of key methodological and analytical information to reviewers and readers." Then make this checklist visible to authors, and provide them with their own checklists to complete for you upon submission: "Require authors to fill out a checklist, ideally upon submission, to state where the required information is located in the manuscript."
Replicates (mentioned here) have not yet been added to this spreadsheet, as unsure if that was too domain specific.
Promoting Reusable and Open Methods and Protocols (PRO-MaP): Draft recommendations to improve methodological clarity in life sciences publications, Leite et al.
Life Sciences
PRO-MaP
Specifically, Section and Table 3 (Publishers and editors) was used in this review.
PRO-MaP does not include study-level protocols or study design. I am not including them here for the reasons stated in PRO-MaP ("Study design protocols include many details that are essential to understand and critically evaluate the study, but are less likely to be reused (e.g. because the study population, inclusion and exclusion criteria are unique to the study)." However, we can include them if required (e.g. with a GUPID).
REAPPRAISED checklist for evaluation of publication integrity
Subject Agnostic, started in Life Sciences
REAPPRAISED
E - Ethics; A - Analysis and Methods;
D - Data duplication and reporting
Much of this does NOT fall under reproducibility checks so hasn't been included. Integrity checks are interesting but out of scope for the reproducibility handbook.
Life Sciences
STORMS
Section 8 (Reproducibility); 16 and 17 (Supplements and supplementary data)
The majority of this checklist is too fine-grained for use within the Handbook.
TOP/COS checklist for editors/reviewers. See also https://osf.io/55eu7 (Checklist for Authors implementing Level 1); https://osf.io/87v93 (Checklist for Editors Levels 1 and 2).
Subject Agnostic
TOP1; TOP2
There are additional fields in these TOP checklists that haven't been included here around replication, pre-registration of analysis plans, and preregistration of studies. All seem out of scope for the more general approach we've been taking. There are also additional requirements around digital object transparency that seem to go into too much detail at this stage.
Excluded
Computer Science
There is some overlap with this work, but broadly speaking this resource is focused on details that are not in scope for us.
Subject Agnostic
Not related to this work at all; discounted
Software/code
No additional information - just an implementation of FAIR4RS
GCCP
Life Sciences
n/a (too specific)
GCCP, GIVIMP, SciRAP GD211
-
n/a (too specific)
Subject Agnostic
Related to the RRPW listed below in the 'possible relevance in future' section
Medical Imaging
reproducibility checklist for authors, to then be used by reviewers etc. However, it too was too narrow in scope and discounted for this review.
Molecular Dynamics
Too narrow in scope; discounted for this review
Possible relevance in the future
Publishers comments at https://docs.google.com/document/d/1B9glUjJRvLvV-QrOdW96WNM8bOglyyPzjZHTVYreFik/edit
I have already reviewed those items but may be worth revisiting in future
Review the questions ARGOS has for RMPs (Elli P)
This paper was read and assessed, but is not directly relevant to this review. It describes an example canonical workflow for publishers to follow when assessing the digital objects associated with manuscript submission. While useful in the context of a workflow, it is not directly relevant to this checklist. Note that one thing publishers mentioned in our meetings is how publisher workflows both vary widely and are often private due to concerns about too much information about checks being released and possibly getting into the hands of paper mills. Integration of our checklist into an example workflow might be one method to use in future as we build the intervention.
Still in development
Other material
Inspiration for the organisational structure of the checklist has come, in part, from: Woodford, C. J., Treloar, A., Leggott, M., Payne, K.., Jones, S., Lopez Albacete, J., Madalli, D., Genova, F., Dharmawardena, K., Chibhira, N., Åkerström, W. N., Macneil, R., Nurnberger, A., Pfeiffenberger, H., Tanifuji, M., Zhang, Q., Jones, N., Sesink, L., & Wood-Charlson, E. (2023). The Global Open Research Commons International Model, Version 1 (Version 1). Research Data Alliance. https://doi.org/10.15497/RDA00097
The 'example' section within each checklist element contains a practical example of how that element is assessed using one of the following manuscripts:
[1] Buljan I, Pina DG and Marušić A. Ethics issues identified by applicants and ethics experts in Horizon 2020 grant proposals [version 2; peer review: 2 approved]. F1000Research 2021, 10:471 (https://doi.org/10.12688/f1000research.52965.2)
[2] Marchant G, Chevance G, Ladino A et al. Behavioural patterns of university students during the COVID-19 pandemic: A cross-sectional study of the effects of active transportation, uninterrupted sitting time, and screen use on physical activity and sitting time [version 3; peer review: 1 approved, 2 approved with reservations]. F1000Research 2025, 11:568 (https://doi.org/10.12688/f1000research.117843.3)
Version updates
Here is a short summary of updates to the site:
3 March 2025: added 13th check and this versioning section.
12 February 2025: updated participants table
16 July 2024: initial release
Last updated