Context, credits and references

Version of 3 March 2025; CC BY-SA 4.0.

Part of the TIER2 project

The Handbook is part of the TIER2 project, enhancing Trust, Integrity and Efficiency in Research through next-level Reproducibility. TIER2 receives funding from the European Union's Horizon Europe research and innovation programme under grant agreement No 101094817.

The Handbook is one of the eight Pilots in TIER2, designed to develop, implement, and evaluate new reproducibility-related tools and practices, emphasising stakeholder engagement and collaboration throughout the project's duration. One of the two pilots targeted to publishers, the Handbook is known in TIER2 as Pilot 8, and it is described here.

A TIER2 workshop with publishers in May 2023 highlighted the ongoing challenge with the sharing of datasets, code, material and other digital objects underpinning results in publications. A consensus priority among the workshop attendees was the need for a common educational and practical set of checks to help in-house editorial staff managing the manuscripts to ensure digital objects were shared in a FAIR-compliant manner and ultimately reproducibile. However, a key requirement was to have a small number of checks, feasible and achievable. An incremental alignment and consistency across journals and publishers was the message, over an attempt to make a comprehensive but unrealistic list of checks, which ultimately will not be put in practice.

In 2024, the academics and professionals in publishing organisations, listed below, worked to co-define and co-create the Handbook via a series of interactive workshops that focused on two workstreams:

1. The Checklist workstream: identifications of the checks, definition, value and implementation; where relevant, using FAIRsharing to signpost standards and data resources, and indicate which checks can be automatised and how.

2. The Workflow workstream: definition of a consensus workflow indicating which roles should perform which checks and when during the internal manuscript process.

Although the primary target audience are in-house editors managing the manuscripts, the Handbook also provide: (i) advice to reviewers, on what compliance to the journal data policy may require, (ii) information for authors on what is expected from them by a number of journals, and (iii) requirements source for developers to drive their service provisions to publishers.

As of February 2025, the intervention is in progress and will run for up to six months, and the results will be published in later this year, along with all underlying data.

Leaders and participants

For more information, please, contact the pilot co-leads.

Name	Affiliation	Title	Role in pilot (and TIER2)
Susanna-Assunta Sansone ([e-mail])	University of Oxford	University Academic Lead of Research Practice; Prof of Data Readiness; OeRC Director	Co-lead (funded partner)
Allyson Lister ([e-mail])	University of Oxford	FAIRsharing Content and Community Coordinator	Co-lead (funded partner)
Rebecca Taylor-Grant ([e-mail])	Taylor & Francis	Head of Data Initiatives	Co-lead (associate partner)
Matt Cannon ([e-mail])	Taylor & Francis	Head of Open Research	Co-lead (associate partner)
Christopher Osborne ([e-mail])	University of Oxford	DPhil Student: improving research culture and practice	Intervention lead (funded student)
Tony Ross-Hellauer	TU Graz	Group Leader	Coord group (TIER2 PI)
Thomas Klebel	TU Graz	Project Manager & Researcher	Coord group (TIER2 Project manager)
Liz Allen		independent expert	TIER2 Advisory Board
Abigail Smith	Wiley	Senior Specialist, Publishing Strategy & Policy Team	Pilot member
Adam Leary	Oxford University Press	Senior Publisher, Open Research	Pilot member
Andrew Hufton	Cell Press	Editor In-Chief (Patterns)	Pilot member
Annie Hill	American Psychological Association	Editorial Director, Journals	Pilot member
Bernd Pulverer	EMBO Press	Head of Scientific Publications	Pilot member
Catriona J. MacCallum	Wiley	Director, Future of Scholarly Research	Pilot member (TIER2 Advisory Board)
Chris Hunter	GigaScience	GigaDB Director	Pilot member
David Tovey	Journal of Clinical Epidemiology / Elsevier	Co-Editor	Pilot member
Elisa De Ranieri	Cell Press	Head of Research Integrity and Author Experience	Pilot member
Grace Ranola	Journal of Petrology / Oxford University Press	Publisher	Pilot member
Guy Jones	Springer Nature	Chief Editor	Pilot member
Hannah Sonntag	EMBO Press	Open Science Implementation Coordinator	Pilot member
Jess Miorini	Language and Cognition	Publisher	Pilot member
Kiera McNeice	Cambridge University Press	Research Data Manager	Pilot member
Lauren Cadwallader	PLOS	Open Research Manager	Pilot member
Libby Dunkley	Cambridge University Press	Editorial Operations Manager	Pilot member
Louie Sandys	Research Directions, Cambridge University Press	Publisher	Pilot member
Monica Moniz	Research Directions, Cambridge University Press	Publisher and Programme Manager	Pilot member
Molly Cranston	F1000 (Taylor & Francis)	Editorial Content Manager	Pilot member
Nicole Nogoy	GigaScience	Executive Editor	Pilot member
Pierre Nauleau	Lancet	Senior Editor	Pilot member
Scott Edmunds	GigaScience	Editor-in-Chief	Pilot member

Existing material revised to identify the checks

A number of documents were reviewed in order to create a consensus Checklist that would work for a variety of publishers and that incorporates the most frequently recommended elements of each source.

Those that were used as sources for the checklist are included in the section below, and referenced within the Guidance documents via the labels from the 'Acronym used' column.nThose that were not used as sources for the Checklist (as they were deemed out of scope) are included in a separated section below.

Included Sources

Document	Research area	Acronym	Sections utilised	Comments
ARRIVE	Life Sciences	ARRIVE	Section 19: Protocol registration; Section 20: Data Access
F1000 checks summary	Subject Agnostic	F1000
FAIR4RS	Software/code	FAIR4RS	F1, F2-F4, R1.1, R3	Excluded as too granular: F1.1 (different granularity getting additional PIDs), F1.2 (different versions get different PIDs), all A principles (dealing with longer-term preservation and resolution of identifiers), all I principles (do journals really check that software interoperates using standard formats?), R1.2 (provenance - out of scope?) and R2 (references to other software)
GigaScience Minimum Standards of Reporting Checklist (was: BMC Better reporting for better research: a checklist for reproducibility)	Life Sciences	GigaScience	Resource subsection; Availability of data and materials subsection	Generally out of scope for this handbook, but some recommendations are at a high-enough level to be included
The MDAR (Materials Design Analysis Reporting) Framework for transparent reporting in the life sciences (see also here and the MDAR checklist and here.)	Life Sciences	MDAR	Specifically, the MDAR checklist for authors. This is also explicitly intended for editors	MDAR checklist items that we are not currently including, as are trying to keep a narrow focus on reusability and reproducibility: study-level protocols, study design, statistical tests, attrition. See also MDAR here: https://www.ncbi.nlm.nih.gov/books/NBK556610/, incl comments about how 'less is more' in terms of checklist items, to increase completeness. This can be partially accomplished through domain-specific formats being used for the digital objects in the first place, and the completeness of the metadata in those formats. That lessens the burden for the editors/reviewers.
Nature Portfolio Editorial Policy Checklist (another link)	Subject Agnostic	Nature	Code and data availability
NIH Principles and Guidelines for Reporting Preclinical Research	Life Sciences	NIH		Note that the NIH states that "Journals should use a checklist during editorial processing to ensure the reporting of key methodological and analytical information to reviewers and readers." Then make this checklist visible to authors, and provide them with their own checklists to complete for you upon submission: "Require authors to fill out a checklist, ideally upon submission, to state where the required information is located in the manuscript." Replicates (mentioned here) have not yet been added to this spreadsheet, as unsure if that was too domain specific.
Promoting Reusable and Open Methods and Protocols (PRO-MaP): Draft recommendations to improve methodological clarity in life sciences publications, Leite et al.	Life Sciences	PRO-MaP	Specifically, Section and Table 3 (Publishers and editors) was used in this review.	PRO-MaP does not include study-level protocols or study design. I am not including them here for the reasons stated in PRO-MaP ("Study design protocols include many details that are essential to understand and critically evaluate the study, but are less likely to be reused (e.g. because the study population, inclusion and exclusion criteria are unique to the study)." However, we can include them if required (e.g. with a GUPID).
REAPPRAISED checklist for evaluation of publication integrity	Subject Agnostic, started in Life Sciences	REAPPRAISED	E - Ethics; A - Analysis and Methods; D - Data duplication and reporting	Much of this does NOT fall under reproducibility checks so hasn't been included. Integrity checks are interesting but out of scope for the reproducibility handbook.
STORMS (another link)	Life Sciences	STORMS	Section 8 (Reproducibility); 16 and 17 (Supplements and supplementary data)	The majority of this checklist is too fine-grained for use within the Handbook.
TOP/COS checklist for editors/reviewers. See also https://osf.io/55eu7 (Checklist for Authors implementing Level 1); https://osf.io/87v93 (Checklist for Editors Levels 1 and 2).	Subject Agnostic	TOP1; TOP2		There are additional fields in these TOP checklists that haven't been included here around replication, pre-registration of analysis plans, and preregistration of studies. All seem out of scope for the more general approach we've been taking. There are also additional requirements around digital object transparency that seem to go into too much detail at this stage.

Excluded

Document name	Research area	Comments
AAAI Reproducibility Checklist	Computer Science	There is some overlap with this work, but broadly speaking this resource is focused on details that are not in scope for us.
Checklist for an Open Research Action Plan	Subject Agnostic	Not related to this work at all; discounted
CONSORT	Life Sciences	n/a (too specific)
FAIR software checklist and tool	Software/code	No additional information - just an implementation of FAIR4RS
GCCP	Life Sciences	n/a (too specific)
GCCP, GIVIMP, SciRAP GD211	-	n/a (too specific)
Limor Peer / RDA / CURE-FAIR	Subject Agnostic	Related to the RRPW listed below in the 'possible relevance in future' section
MICCAI Reproducibility Checklist	Medical Imaging	reproducibility checklist for authors, to then be used by reviewers etc. However, it too was too narrow in scope and discounted for this review.
PRISMA	Life Sciences	n/a (too specific)
Reliability and reproducibility checklist for molecular dynamics simulations	Molecular Dynamics	Too narrow in scope; discounted for this review

Possible relevance in the future

Document name	Comments
Publishers comments at https://docs.google.com/document/d/1B9glUjJRvLvV-QrOdW96WNM8bOglyyPzjZHTVYreFik/edit	I have already reviewed those items but may be worth revisiting in future
Review the questions ARGOS has for RMPs (Elli P)
Reproducible Research Publication Workflow: A Canonical Workflow Framework and FAIR Digital Object Approach to Quality Research Output	This paper was read and assessed, but is not directly relevant to this review. It describes an example canonical workflow for publishers to follow when assessing the digital objects associated with manuscript submission. While useful in the context of a workflow, it is not directly relevant to this checklist. Note that one thing publishers mentioned in our meetings is how publisher workflows both vary widely and are often private due to concerns about too much information about checks being released and possibly getting into the hands of paper mills. Integration of our checklist into an example workflow might be one method to use in future as we build the intervention.
Enhancing Peer Review Standards for Quality Research Publications	Still in development

Other material

Inspiration for the organisational structure of the checklist has come, in part, from: Woodford, C. J., Treloar, A., Leggott, M., Payne, K.., Jones, S., Lopez Albacete, J., Madalli, D., Genova, F., Dharmawardena, K., Chibhira, N., Åkerström, W. N., Macneil, R., Nurnberger, A., Pfeiffenberger, H., Tanifuji, M., Zhang, Q., Jones, N., Sesink, L., & Wood-Charlson, E. (2023). The Global Open Research Commons International Model, Version 1 (Version 1). Research Data Alliance. https://doi.org/10.15497/RDA00097

The 'example' section within each checklist element contains a practical example of how that element is assessed using one of the following manuscripts:

[1] Buljan I, Pina DG and Marušić A. Ethics issues identified by applicants and ethics experts in Horizon 2020 grant proposals [version 2; peer review: 2 approved]. F1000Research 2021, 10:471 (https://doi.org/10.12688/f1000research.52965.2)

[2] Marchant G, Chevance G, Ladino A et al. Behavioural patterns of university students during the COVID-19 pandemic: A cross-sectional study of the effects of active transportation, uninterrupted sitting time, and screen use on physical activity and sitting time [version 3; peer review: 1 approved, 2 approved with reservations]. F1000Research 2025, 11:568 (https://doi.org/10.12688/f1000research.117843.3)

Version updates

Here is a short summary of updates to the site:

• 3 March 2025: added 13th check and this versioning section.

• 12 February 2025: updated participants table

• 16 July 2024: initial release