Systematic Reviews: About Systematic Reviews

Given the prolific nature of Academic Publishing, it can be very difficult to come to a conclusion based on single research papers, no matter how high-impact or widely read. This is especially important in the field of Medicine where high quality evidence of the benefits and risk of treatments is literally a matter of life and death.

Well-funded studies with large sample sizes but with possible evidence of bias may show one conclusion (for example the prescribing of a certain drug) while smaller independent studies may show possible evidence of harm of the same drug. These studies need to be looked at together, or we risk:

• Missing risks and benefits, even slight which may be surfaced by combining multiple studies. 
• Never having a clear evidence-based conclusion, and relying on personal opinion, politics or personalities.
• Falling prey to Confirmation Bias, where only studies that align with a persons prevailing opinions or beliefs are included in evidence reviews or guidelines.
• Wasting money funding research that may have not exactly been done before, but that clear evidence is surfaced by combining similar studies that shows the research may not be justified.

Some recommended additional reading

Editorial: What Makes Systematic Reviews Systematic and Why are They the Highest Level of Evidence 

YouTube: Mixed Results in Sets of Studies (Importance of systematic review) 

YouTube: Introduction to Meta-analysis (Importance of meta-analysis) 

Cochrane: Evidence Synthesis - What is it and why do we need it?

Doing A Literature Review In Health And Social Care - The Systematic Review (Lancaster Subscription)

Systematic Reviews Leganto Resource List

Systematic Review Teams Channel

A good starting point for understanding the need for Systematic Reviews, and start your journey towards choosing a topic is to understand about "uncertainty". Topics chosen for a Systematic Review are not chosen out of general interest or curiosity, but due to an inherent uncertainty in the literature, meaning no clear conclusion or evidence exists for one course of action over another. Uncertainty can be found in:

• Literature Searching:  A well performed systematic literature search providing no consensus opinion.
• The existence of possibly harmful pseudoscientific methods : A prevailing belief in a certain course of action without clear evidence, for example the consumption of a common folk remedy for seasonal flu.
• Difficulty identifying the best technologies for a specific problem due to possible bias or entrenched views, for example the best programming language to use for rapid code deployment.
• Conclusions from well run systematic reviews that state uncertainty exists: It is common that systematic reviews will state that certain uncertainties exist, and by reading Systematic Reviews in your subject area you may discover possible topics for review.
• Experience: You may be confronted by differing opinions of one course of action over another, and wish to create an evidence-based, strongly compelling and independent guidance document.

Boolean

A type of algebra used in database searches. Usually you would use AND / OR searching and avoid the NOT function in a Systematic Review

Heterogeneity

Are all the studies in a systematic review similar? Then it would be stated that the paper has 'Low Heterogeneity. If the studies included in a systematic review differ due to research methodologies or reporting methods, then it has possibly 'High Hetereogenity' and it would be difficult to meaningfully compare them.  

Highly Sensitive Search Strategy (HSSS)

A search strategy that is highly sensitive, missing as few relevant papers as possible with the drawback of an increased number of irrelevant papers.

Inclusion/Exclusion/Eligibility Criteria

What papers you are Including in your Systematic Review (via your Systematic search) and What you are excluding (Via Screening). 

Meta-Analysis and Synthesis

A statistical method of combining evidence, or results of studies. A systematic review is refered to a type of 'Evidence Synthesis' and the Meta-Analysis can also be called 'Evidence Synthesis. Not all systematic reviews have a Meta-Analysis, and it may be unsuitable in mixed-methods and non-health Systematic Reviews.

Protocol

The writeup of the methodology of the Systematic Review and detaining the planning, search strategy and methods. It should be prepared before commencement of a Systematic Review and referred to for guidance.

PRISMA 2020

PRISMA is a minimum set of items for reporting in systematic reviews and meta-analyses. Typically used for interventions in medicine, it is one of the most popular protocols to adapt to other non-clinical subject areas. The PRISMA flow diagram is in extensive use as a visual representation of a literature search.

Proximity Operators

An operator used in a database search. The most commonly used operator is 'near' which retrieves papers if words are within a defined number of each other. In EBSCO databases it is N3 while in OVID it is ADJ3. 

.RIS

A standard document format for exporting references from database searches, and then importing into reference management software or screening applications/websites.

Reproducibility

The ability to easily reproduce a literature search strategy in the future

Search Strategy

The 'strategy' for your systematic search. This should be based if possible on a methodology that retrieves the most relevant papers possible with a risk of some non-relevant papers included. The aim is to reduce the number of papers to a point that can be pragmatically screened, not produce a search that limits to a small number of highly relevant papers!

It may be that a traditional Systematic Review is not suitable for all questions.

• Poor evidence base: A systematic review relies on a firm evidence base, and if a subject does not have sufficient literature it may be impossible to complete. This may be common in very new technologies. A scoping search is suitable before deciding on your systematic review topic to ensure sufficient evidence.

• Lack of time, experience or support: Producing a Systematic Review is a time-consuming process, and the quality of the review is strongly influenced by the skills and experience of your review team. Considering consulting a Faculty Librarian in your team if you feel that additional assistance may be needed, or contribute to other Systematic Reviews in order to gain experience before attempting your own.

• Risk of high levels of Heterogeneity: In some types of Systematic Review (especially in health in a SR with Meta-Analysis) to be suitable it should show a low level of 'heterogeneity'. High heterogeneity is where included studies differ from one another significantly in the study design and evaluation, making it difficult to evaluate one paper against another. In some subjects this is less important

However: If you find either a poor evidence base, zero evidence, or significant Heterogenity, the reasons for this can be studied in further research.
 

Congratulations!

This guide is split into two parts, specific guidance pertaining to Health, and also guidance focusing on Science and Technology SRs.

You can then visit the systematic review 'Pathway' with a recommended general process and useful links called 'Undertaking your Systematic Review'.

If you have decided that a systematic review may not be the most suitable course of action, you may wish to consider

Rapid Review

A review type usually performed in a shorter time period, and often used for new and emergent topics. Has some elements of a Systematic Review, but with limits on time, not all. Review the Cochrane Rapid Reviews Methods Group paper for further information.

Umbrella Review

With the popularity of Systematic Reviews some subjects may already have multiple published reviews. An umbrella review is a review of previously published Systematic Reviews with a uniform treatment of meta-analysis across all reviews. These form one of the most rigorous and highest levels of evidence synthesis. Read  Ten simple rules for conducting umbrella reviews for a summary.

Systemized Review:

Many systematic reviews which are produced on a sole basis in small teams could accurately be referred to as Systemized Reviews. They may lack some of the elements of a Systematic Review such as a meta-analysis, however still be produced with elements of a Systematic Review.

Scoping Review

A type of review that aims to identify the scope of literature on a topic. Useful to identify the feasibility of a systematic review. Read Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach for a summary.

Integrative Review

A review type that combines diverse methodologies such as experimental and non-experimental research. Read ‘The integrative review: updated methodology’

Executive Evidence Summary

A piece of work not intended for journal publishing, but for the attention of organisational or national bodies in setting policy or guidance. These can be produced in a manner similar to a systematic review to create a highly compelling and evidence-based work.

Systematic Literature Review

A summary of the literature on a topic. It may be that either due to time constraints, qualitative nature of the topic or possible high levels of heterogeneity in studies,  a systematic review with meta-analysis isn't possible to complete. A literature review completed to the high standards of a systematic review, perhaps without a meta-analysis may be suitable. 

Rapid Evidence Appraisal (REA)

More rigorous than a literature review. REAs summarise what is and is not known in a subject, with elements of a systematic review included that minimise bias.  Designed to be quicker than a systematic review.

Realist Review

While Systematic reviews focus on a specific topic, realist reviews represent a theory-based approach to synthesis. This type of review can require a greater commitment of time working with an Information Professional or Librarian, so ensuring you have access to such staff is key. Read Scoping and Searching to Support Realist Approaches in Doing Realist Research (Sage Research Methods Core Subscription)

Note that many of these review types are very similar! "Almost" systematic reviews in different disciplines such as sociology or computing may be called different names while sharing tell-tale methodologies such as a published protocol, search strategy reporting or Prisma flow diagrams.

The paper A typology of reviews: an analysis of 14 review types and associated methodologies has an extensive list of the strengths and weaknesses of different review types.

This guide is available (CC BY-NC 3.0)

If you do reuse, please let us know so we can share any updates.