Registered Reports

Registered Reports are a publication format in which the research question and the quality of methodology are peer reviewed before the data are collected and analysed. High quality protocols addressing well framed questions are then provisionally accepted for publication before data collection begins. This format allows methodological issues to be addressed before time and resources are invested in experiments, and helps minimize publication bias and research bias in hypothesis-driven research.

The format also however allows exploratory (unregistered) analyses that may be suggested by interim results, provided that the Editor is notified;  and the inclusion of serendipitous findings.

The submission and review process for Registered Reports is divided into two distinct stages.

  • Stage 1 submission: Authors submit manuscripts including only an Introduction, Methods (including proposed analyses), and Pilot Data (where applicable).
  • Stage 1 review: Reviewers evaluate study proposals before data are collected, assessing the importance of the research question, feasibility of the methodology, and analysis pipeline.
  • In principle acceptance (IPA): Manuscripts that pass peer review will be issued an IPA, indicating that the article will be published pending successful completion of the study.
  • Stage 2 submission: Following study completion, authors submit their finalized manuscript for re-review, now including Results and Discussion sections.
  • Stage 2 review: Reviewers appraise whether the authors’ adhered to the preregistered experimental procedures and that any conclusions are justified given the data.

Information for Authors

Essential feature of the Registered Reports format is that a significant part of the manuscript is assessed prior to data collection. The submission and review process is consequently divided into two stages.

Stage 1: Initial manuscript submission and review

Initial submissions should include the stage 1 manuscript (requirements detailed below) and a brief cover letter. Authors are welcome to first submit presubmission enquiries for advice on the likely suitability of a study as a Registered Report.

Cover letter

Please include a brief scientific case for consideration and an estimated timeline for completing the study if the initial submission is accepted in principle.

Introduction

This section should include a review of the relevant literature that motivates the research question and a full description of the experimental aims and hypotheses. Please note that following IPA, the Introduction section cannot be altered.

Methods

The methods section should include:

  • Full description of proposed sample characteristics, including criteria for data inclusion and exclusion (e.g. outlier extraction). Procedures for objectively defining exclusion criteria due to technical errors or for any other reasons must be specified, including details of how and under what conditions data would be replaced.
  • A description of experimental procedures in sufficient detail to allow another researcher to repeat the methodology exactly, without requiring further information. These procedures must be adhered to exactly in the subsequent experiments or any Stage 2 manuscript can be rejected. 
  • Proposed analysis pipeline, including all preprocessing steps, and a precise description of all planned analyses, including appropriate correction for multiple comparisons. Any covariates or regressors must be stated. Where analysis decisions are contingent on the outcome of prior analyses, these contingencies must be specified and adhered to. Only pre-planned analyses can be reported in the main Results section of Stage 2 submissions. However, unplanned exploratory analyses will be admissible in a separate section of the Results (see below).
  • Studies involving Neyman-Pearson inference must include a statistical power analysis. Estimated effect sizes should be justified with reference to the existing literature. Since publication bias overinflates published estimates of effect size, power analysis must be based on the lowest available or meaningful estimate of the effect size. For frequentist analysis plans, the a priori power must be 0.9 or higher for all proposed hypothesis tests. In the case of highly uncertain effect sizes, a variable sample size and interim data analysis is permissible but with inspection points stated in advance, appropriate Type I error correction for ‘peeking’ employed, and a final stopping rule for data collection outlined.
  • Methods involving Bayesian hypothesis testing are encouraged. For studies involving analyses with Bayes factors, the predictions of the theory must be specified so that a Bayes factor can be calculated. Authors should indicate what distribution will be used to represent the predictions of the theory and how its parameters will be specified. For example, will you use a uniform up to some specified maximum, or a normal/half-normal to represent a likely effect size, or a JZS/Cauchy with a specified scaling constant? For inference by Bayes factors, authors must be able to guarantee data collection until the Bayes factor is at least 6 times in favour of the experimental hypothesis over the null hypothesis (or vice versa). Authors with resource limitations are permitted to specify a maximum feasible sample size at which data collection must cease regardless of the Bayes factor; however to be eligible for advance acceptance this number must be sufficiently large that inconclusive results at this sample size would nevertheless be an important message for the field. 
  • Full descriptions must be provided of any outcome-neutral criteria that must be met for successful testing of the stated hypotheses. Such quality checks might include the absence of floor or ceiling effects in data distributions, positive controls, or other quality checks that are orthogonal to the experimental hypotheses.
  • Timeline for completion of the study and proposed resubmission date if Stage 1 review is successful. Extensions to this deadline can be negotiated with the action editor.
  • Any description of prospective methods or analysis plans should be written in future tense

Pilot Data

  • Optional. Can be included to establish proof of concept, effect size
    estimations, or feasibility of proposed methods. Any pilot experiments will be
    published with the final version of the manuscript and will be clearly
    distinguished from data obtained for the pre-registered experiment(s). 

Following stage 1 peer review manuscripts will either be rejected, given the opportunity to revise or offered an in-principle acceptance (IPA). An IPA indicates that the article will be published pending completion of the approved methods and analytic procedures, in addition to a defensible interpretation of the results.

Authors are reminded that any deviation from the stated experimental procedures, regardless of how minor it may seem to the authors, could lead to rejection of the manuscript at Stage 2. In cases where the pre-registered protocol is altered after IPA due to unforeseen circumstances (e.g. change of equipment or technical error), the authors must consult the Editors immediately for advice, and prior to the completion of data collection. Minor changes to the protocol may be permitted according to editorial discretion. In such cases, IPA would be preserved and the deviation reported in the Stage 2 submission. If the authors wish to alter the experimental procedures more substantially following IPA but still wish to publish their article as a Registered Report then the manuscript must be withdrawn and resubmitted as a new Stage 1 submission.

Note that registered analyses must be undertaken, but additional unregistered analyses can also be included in a final manuscript (see below).

Stage 2: Full manuscript submission and review

Authors will be asked to complete their study and resubmit their manuscripts within 12 months of receiving an in-principle acceptance. Failure to meet the agreed timetable may result in a stage 1 manuscript being considered withdrawn.

A stage 2 submission should include the following additions:

Cover Letter                                          

  • Confirmation in the resubmission Cover Letter that all non-pilot data was collected after the data of IPA, and analysed in the manner previously described. Any unforeseen changes in methods and analyses must be clearly noted.
  • Please note that the Introduction cannot be altered from the approved stage 1 submission, and the stated hypothesis cannot be amended or expanded.

Results & Discussion

  • The outcome of all registered analyses must be reported in the manuscript, except in rare instances where a registered and approved analysis is subsequently shown to be logically flawed or unfounded. In such cases, the authors, reviewers, and editor must agree that a collective error of judgment was made and that the analysis is inappropriate. In such cases the analysis would still be mentioned in the Methods but omitted with justification from the Results.
  • It is reasonable that authors may wish to include additional analyses that were not included in the registered submission. For instance, a new analytic approach might become available between IPA and stage 2 review, or a particularly interesting and unexpected finding may emerge. Such analyses are admissible but must be clearly justified in the text, appropriately caveated, and reported in a separate section of the Results titled “Exploratory analyses”. Authors should be careful not to base their conclusions entirely on the outcome of statistically significant post hoc analyses.
  • Authors reporting null hypothesis significance tests are required to report exact p-values and effect sizes for all inferential analyses.
  • Raw and processed date must be made freely available in accordance with our data sharing policies.

Please see our Registered Reports submission guidelines for details of the additional Declarations that must be included, as with standard article types

Manuscript withdrawal

Manuscripts can be withdrawn at the authors’ discretion following in-principle acceptance. However, the journal will publish the manuscript’s proposed title, author names, the abstract from the approved stage 1 submission and a brief reason for the failure to complete the study.

 

 

 

 

 

 

 

Information for Reviewers

This guide for reviewers contains information on the peer-review process for Registered Reports. If you have been invited to review for the journal and have any queries, please do not hesitate to contact our Editorial Team.

The review process for Registered Reports is divided into two stages. At stage 1, reviewers are asked to assess study proposals before data are collected. At stage 2, reviewers consider the full study, including results and interpretation.

Stage 1: Initial manuscript submission and review

Stage 1 manuscripts will include only an Introduction, Methods (including proposed analyses), and Pilot Data (where applicable). In considering papers at stage 1, we ask reviewers to consider assess:

  1. The importance of the research question(s).
  2. The logic, rationale, and plausibility of the proposed hypotheses.
  3. The soundness and feasibility of the methodology and analysis pipeline (including statistical power analysis where appropriate).
  4. Whether the clarity and degree of methodological detail is sufficient to exactly replicate the proposed experimental procedures and analysis pipeline.
  5. Whether the authors have pre-specified sufficient outcome-neutral tests for ensuring that the results obtained are able to test the stated hypotheses, including positive controls and quality check.

Following stage 1 peer review, manuscripts will be accepted, offered the opportunity to revise, or rejected outright. Manuscripts that pass peer review will be issued an in principle acceptance (IPA), indicating that the article will be published pending successful completion of the study according to the pre-registered methods and analytic procedures, as well as a defensible and evidence-based interpretation of the results. Authors will be asked to complete their study within 12 months of IPA.

Stage 2: Full manuscript submission and review

Following completion of the study, authors will complete the manuscript, including Results and discussion sections. These stage 2 manuscripts will more closely resemble a regular article format. The manuscript will then be returned to the reviewers, who will be asked to appraise:

  1. Whether the data are able to test the authors’ proposed hypotheses by satisfying the approved outcome-neutral conditions (such as quality checks, positive controls).
  2. Whether the Introduction, rationale and stated hypotheses are the same as the approved stage 1 submission (required).
  3. Whether the authors adhered precisely to the registered experimental procedures.
  4. Whether any unregistered post hoc analyses added by the authors are justified, methodologically sound, and informative.
  5. Whether the authors’ conclusions are justified given the data.

Reviewers at stage 2 may suggest that authors report additional post hoc tests on their data; however authors are not obliged to do so unless such tests are necessary to satisfy one or more of the stage 2 review criteria. Please note that editorial decisions will not be based on the perceived importance, novelty, or conclusiveness of the results.