Work package 4

Statistical modeling and software to assess the impact of unpublished studies on meta-analysis

Objectives

Design, development, and testing of open-source software that will utilize the available information from unpublished studies in trial registries to estimate the potential impact of these studies on the results of a given meta-analysis

The new software will assist researchers in conducting meta-analyses to gauge the potential impact of missing data of ongoing and unpublished studies on their results and conclusions in meta-analyses.

Task 4.1 - Background literature overview

Background literature searches and evaluation of literature on modelling to assess the impact of missing data on random and fixed effects meta-analyses.

Task 4.2 - Development of Dataset and code

• Development of a hypothetical dataset for meta-analyses on a dichotomous outcome and a variety of hypothetical unpublished studies
• Writing of the code using Markov Chain models to assess the necessary magnitudes of treatment effects in given (but hypothetical) unpublished studies to lead to relevant changes of results in a random or fixed effects meta-analysis
• Three relevant scenarios will be explored by the new software:
  • Magnitude of effect in unpublished studies to change the status of statistical significance in the pooled estimate of the meta-analysis (either from statistical significance to non-significance, or vice versa)?
  • Magnitude of effect in unpublished studies to reverse the direction of the effect in the pooled estimate of the meta-analysis (e.g. from a result favouring intervention A over intervention B to a result favouring intervention B over intervention A)? 
  • Magnitude of effect in unpublished studies to change the status of clinical relevance in the pooled estimate of the meta-analysis (either from clinical relevance to irrelevance, or vice versa)?

Task 4.3 - Finalization of program code

• Trials and tests of the code across various scenarios of meta-analyses and unpublished studies will be run, e.g., based on data described in WP2
• Revision of the code based on results and usability.

Software

A software package, called SAMURAI (Sensitivity Analysis of a Meta-analysis with Unpublished but Registered Analytical Investigations) was developed using the open source statistical programming language R, using standard meta analytic procedures in R. Users may not be familiar with using R, so the interface required to access the program is Microsoft Excel, which is widely available. The developed software was extensively tested, and it is delivered here along with a User’s Guide and sample data sets.

The software includes the following files: