As per Cochrane definition (2013), a systematic literature review (SR) attempts “to identify, appraise and synthesize all the empirical evidence that meets pre-specified eligibility criteria to answer a given research question” [1]. A meta-analysis (MA) is a statistical assessment of the data provided by multiple studies that attempt to ask / answer the same question. SR involves formulated questions, identifies relevant studies, appraises their quality, and summarises the evidence by use of explicit methodology.
Systematic reviews are different from narrative reviews, which are more descriptive, focusing typically on a subset of studies in an area of interest. In this article, we will focus on a stepwise approach to SR writing.
Step 1: Research question and objectives
This is the most important aspect of SR / MA. The research question should be novel, interesting, feasible, ethical, and relevant. However, there are two important aspects – firstly, the problem to be addressed should be clear, unambiguous, and structured, and secondly, a thorough search should be completed to be aware of any similar SR / MA published in recent times. Once ready, the question posed can be structured in PICO format – Population, Intervention, Comparison, and Outcome (Table 1).
Step 2: Literature search
Once the research question has been finalised, there is a need for a robust literature search. The easiest approach is to get help from your library. Most libraries can provide literature searches using their available software to search in different databases (like PubMed, Medline / OvidSP (includes EMBase), Web-of-Science, and BIOSIS, etc.). The library requires an online form completion which usually ask for information about the research question, timeline, inclusion and exclusion criteria, and a few appropriate references. Most libraries do an extensive literature search and provide results in various formats and often links with full access. The libraries provide a search strategy as well. This is a useful supplementary document for submission.
Alternatively, a literature search can be performed by authors on their own. After finalising the search question, appropriate keywords / MeSH (Medical Subject Headings) should be identified for each theme. Following this, appropriate Boolean operators, which are simple words including AND, OR, NOT, or AND NOT, are used as conjunctions to combine or exclude keywords in a search, resulting in more focused and productive results. This essentially saves the purpose of eliminating inappropriate hits. All possible electronic databases must be searched. For Cochrane review, a minimum of three databases need to be searched. Appropriate search filters such as duration, language, type of study (animal or human), etc. are required.
Figure 1: Screenshot of Rayyan software.
Step 3: Screening and selection of publications
Study screening and selection is a challenge, which can be made easy by using software like Rayyan [2], illustrated in Figure 1. It helps researchers to work remotely and collaborate within a distributed research team. The software requires literature searches from the library to be uploaded, and authors will need to ask the library to send the search results in a format compatible with the software. Once this has been uploaded authors can include or exclude studies based on inclusion and exclusion criteria. The full-text versions of studies should be sought out for inclusion in the review. Again, library services can be helpful, and if unavailable, direct polite communication with the original author can be helpful.
Figure 2: Screenshot of Mendeley desktop.
All essential collected data should be documented in an appropriate format that suits the author best. Most authors document data in Microsoft Excel and retain references in Mendeley software [3], which is displayed in Figure 2. Other reference managing software options are Zotero and Endnote.
This is the ideal time to complete your Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram (2020) [4]. This should not be confused with the PRISMA checklist, which is usually completed towards the end and often uploaded as a supplementary document.
Step 4: Registration of research protocol
At this stage the study protocol should be registered with the different registry sites recommended by the Cochrane and Campbell collaborations. We recommend registering the research protocol with the International Prospective Register of Systematic Reviews (PROSPERO) [5]. A brief description of registration with the PROSPERO database is provided in Table 2.
Step 5: Interpreting data
We recommend following a two-stage approach. Firstly, tabulate all the studies describing the authors, primary aim, type of study (prospective, retrospective), methodologies used, results, and conclusions in a summarised manner. This table needs to be written in author’s words to make it relevant to SR and avoid plagiarism. This table does go into the final paper. Secondly, author’s should make a master table (preferably Excel or Word) and extract as much information as possible. Even if, in some studies, you feel you do not need additional data, we recommend extracting it as well. This is the most critical part of SR and authors need to be careful and vigilant in interpreting data, as this forms the essential source of information for SR. You can keep modifying the data table as the studies included may have presented data in different ways. Once this master table is ready, author’s should start collating data in new tables as per the aims and objectives of the SR.
Step 6: Risk of bias assessment
Systematic reviews are also susceptible to bias that arise in primary studies, each of which needs to be critically appraised. The Cochrane manual defines risk of bias as “a systematic error, or deviation from the truth, in results.” It further states that “biases can lead to under-estimation or over-estimation of the true intervention effect and can vary in magnitude: some are small (and trivial compared with the observed effect) and some are substantial (where an apparent finding may be due entirely to bias).”
Popular risk of bias assessment tools
- Modified Downs and Black Checklist
Provides both an overall score for study quality and a numeric score out of a possible 30 points. It has five sections. Administration of the tool can be either within the systematic review process or as a quality assessment tool for individual articles. - Newcastle-Ottawa Quality Assessment Scale
A tool used to assess the quality of non-randomised studies. It allocates a maximum of nine stars for the quality of selection, comparability, exposure, and outcome of study participants [6]. - Cochrane Risk of Bias Tool (RoB 2)
Tool used to assess the risk of bias for randomised controlled trials. Required for all Cochrane group systematic reviews. - Cochrane Risk of Bias in Non-Randomised Studies of Interventions (ROBINS-I Tool)
The ROBINS-I is a tool developed to assess the risk of bias in the results of non-randomised studies that compare the health effects of two or more interventions. - MMAT (Mixed Methods Appraisal Tool)
The MMAT is intended to be used as a checklist for concomitantly appraising and / or describing studies included in systematic mixed studies reviews (reviews including original qualitative, quantitative, and mixed methods studies) when the SR includes data from animal research, the Systematic Review Centre for Laboratory Animal Experimentation (SYRCLE)’s risk of bias tool [7] should be used.
Step 7: Analytic choices in case of meta-analysis
Meta-analysis is a quantitative, formal, epidemiological study design used to systematically assess previous research studies to derive conclusions about that body of research. The benefits of meta-analysis include a consolidated and quantitative review of a large, and often complex, sometimes apparently conflicting, body of literature.
Four meta-analytical methods are primarily used in contemporary management research which allow the investigation of different types of research questions:
a. Univariate meta-analysis
b. Meta-regression
c. Meta-analytic structural equation modelling
d. Qualitative meta-analysis.
Software solutions to perform meta-analyses range from built-in functions or additional packages of statistical software to those that are purely focused on meta-analyses. These may be commercial or open-source solutions. However, in addition to personal preferences, the choice of the most suitable software depends on the complexity of the methods used and the dataset itself. Analysts therefore must carefully check if their preferred software is capable of performing the intended analysis.
Among commercial software providers, Stata (from version 16 onwards) offers built-in functions to perform various meta-analytical analyses or to produce various plots. Researchers using the open-source software R, provide an overview of 63 meta-analysis packages and their functionalities. For new users, they recommend the package metafor, which includes the most necessary functions. In addition to packages and macros for statistical software, templates for Microsoft Excel has also been developed to conduct simple meta-analyses such as meta-essentials. Finally, programs purely dedicated to meta-analysis also exist, such as Comprehensive Meta-Analysis or RevMan by The Cochrane Collaboration (2020).
Sometimes due to the nature of data, a network meta-analysis becomes essential to arrive at conclusions. Network meta-analysis is a technique for comparing three or more interventions simultaneously in a single analysis by combining both direct and indirect evidence across a network of studies.
In the end, if the authors find it difficult to perform a statistical analysis due to complex data or lack of knowledge, it is recommended to get help from a professional statistician.
Step 8: Drafting the paper
Once all this is ready, it is time to write the initial draft. The paper should be written either by keeping in focus on a particular journal or being flexible in terms of which journal it would be submitted to for publication. If authors consider a particular journal, it may be worth contacting the editorial team first as some journals entertain only invited reviews. Table 3 is a useful guide to writing the paper. At this stage, the PRISMA checklist should be completed.
Step 9: Article submission
There are a few things to consider before submitting a paper and which must be completed before, including:
a. Author details, affiliations, preferably ORCID numbers
b. Author contribution (check the journal requirements)
c. Acknowledgements (if any)
d. Running title
e. Keyword list
f. Conflict of interest statement
g. Copyright form (specific for each journal)
h. Cover letter.
References
1. https://www.cochranelibrary.com/cdsr
2. https://www.rayyan.ai
3. https://www.mendeley.com
4. http://prisma-statement.org/prismastatement/flowdiagram.aspx
5. https://www.crd.york.ac.uk/prospero
6. https://www.ohri.ca/programs/clinical_epidemiology/oxford.asp
7. Hooijmans CR, Rovers MM, de Vries RB, et al. SYRCLE’s risk of bias tool for animal studies. BMC Med Res Methodol 2014;14:43.
All links last accessed April 2024.
Declaration of competing interests: None declared.
