Welcome to the Systematic Review 'Pathway', a summary step-by-step to introduce you to the various steps involved in taking your Systematic Review through your to your literature search and screening.
Before you book an appointment with a Faculty Librarian for assistance, we encourage you to look at the various steps of the pathway, as many of the most common questions we receive are included.
Also join the Systematic Review teams channel and ask your questions there, for a quick response to your query.
What this pathway does not include?
For the moment we will not be including advice beyond your Systematic Search (Such as Meta-analysis) as support is available in departments.
A scoping search can be done for two reasons:
Before your search.
It is recommended that you create a scoping search through all your databases before you produce your actual search. (like a test run).
This will allow you to 'scope' out any subject terms, terminology, and limitations of each database. If you produce a document detailing your findings, you can also use this in the protocol or 'search strategy' section of your paper as a reflective discussion of the search.
Before starting your Systematic Review
It is important to identify if a recent Systematic Review has been performed on the topic:
A well-produced systematic review should always be considered to be a comprehensive treatment of a topic, therefore it is wise to identify if another Systematic Review has been published. Review the Lancaster Library’s Q&A: How to find published or prospective systematic reviews for advice.
The scoping search can also be useful to identify topics for a systematic review? (Gap Analysis). Sometimes it is difficult to even consider topics suitable for a systematic review of evidence synthesis. The Library FAQ How do I use library databases to identify topics for research or synthesis? provides an example search string that can help you filter a search for possible research topics.
What now?
Found a recent systematic review: Select another topic. Perhaps a similar but distinct topic so you can reference and follow the protocol and search strategy in the paper, saving time.
Found an older systematic review: Consider an update of the paper, following the original search strategy and protocol, and possibly improving it with new frameworks and guidelines.
Found no systematic review: Great news, your paper will likely have a high impact! Make sure that you have retrieved sufficient papers for inclusion, however, as it may be difficult to produce a systematic review on an area with scant or extremely low-quality research.
Before you start your search, read the Systematic Literature Searching LibGuide, and possibly attend library training. We will just discuss elements of a literature search unique to a Systematic Review.
Consider if a search of Grey Literature will be included, or if the scope is worldwide in nature. If so the Grey Literature and Decolonising Literature Searching LibGuide may be of help.
Creating a Question
An excellent start is to use a predetermined framework to create your question. Please remember that these are 'frameworks' and might not work for every search. Common frameworks are PICO and SPIDER. The Systematic Literature Searching LibGuide has a section on defining your question.
If your research question doesn't quite fit (common in science and technology subjects), and you require some help with producing an answerable question consider a booking with a Faculty Librarian, where a mediated search through our databases and a few suitable papers can usually assist in narrowing down a research question.
The search is arguably the most important element of a well-produced systematic review. A good search should be:
Highly Sensitive, it should retrieve the most papers for your question, with the risk of retrieving irrelevant papers. These papers should be removed via screening.
Reproducible, it should be well documented and able to be repeated
Tested, to ensure that key papers have not been overlooked, or key subject terms missed.
Many protocols also recommend that a search is produced in collaboration with an information professional such as a skilled researcher or librarian.
Review the Systematic Literature Searching LibGuide for a more comprehensive guide on this topic, as we will only cover elements related to Systematic Reviews. An excellent paper to review is the PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement from the Journal of Clinical Epidemiology, which includes a handy 'best practice' checklist.
The Below example is also a very summarised version of the comprehensive Cochrane Highly Sensitive Search Strategy
Important!
At every stage of your search, note down the database and the number of results found. This will make producing a PRISMA flow diagram easier. Export your results as a .RIS file. Also, drop your search strategy into a word document (a simple copy and paste works).
This will help you document your search strategy, and ensure reproducability
The guidance How do I export a large number of results from EBSCO databases to EndNote? may help streamline the exporting of abstracts.
Key elements of a Highly Sensitive Search
In a basic HSSS, you will always see these elements (They can be complex in many cases with proximity searching and boolean, but the search tends to follow this format)
The top search box is the Subject Term. Always search through the subject term/thesaurus and add any relevant subject terms. If you are given the option to 'explode' (include all the subheadings) always consider this. You can choose not to but document your reasons why.
The next two boxes are set to search through the Title and Abstract. Normally this is the default search however if a subject is very new you can search 'all fields' as it might bring in a few additional author keywords. This search is a 'free text' search and features a few elements common to HSSS searches:
Brackets. These ensure that search strings inside the brackets are performed first, almost like a mini search within a search. Use these to ensure your search string does not do anything unexpected.
Plurals. You can add a * to also retrieve plurals and other words 'starting with'. If you are curious that adding a * to your word might retrieve irrelevant papers you can use a crossword solver website such as WordHippo to investigate words 'starting with'.
Proximity/Adjacent Operator. This is a special operator such as N, ADJ or NEAR (Depends on database which you can use to search for search terms within N words of each other. This allows you to search for words in proximity with each other, I.E. Health N3 Literat* will retrieve "health literacy", "literate health worker" and "healthy and literate". See the Systematic Searching LibGuide for further information.
Although useful in Quantitative research topics, Proximity searching can be very useful in Qualitative subjects dealing with conceptual topics that might have the same research subject described in many different ways. You can increase the N or ADJ operator to take into account qualitative topics (feelings and opinions) that may not be in as close proximity to each other. A good starting point is N3 for Quantitative and N5 for Qualitative topics. See LibAnswers
A common question: How many papers should I include in my systematic review?
The answer is, as many as your search retrieves! You should not limit by date or any search terms unless they are specifically mentioned in your protocol.
Systematic reviews with 1000s of papers for screening are not unheard of, and should not be artificially reduced. In many cases, a Faculty Librarian may be able to help advise you on improving your search without reducing the sensitivity of the search.
It is also possible to set up Endnote or use services such as Rayyan to streamline the screening process, in a multi-stage 3 pass method such as title, abstract, and full-text screening which reduces the burden.
Often asked: Am I permitted to use a Date range in a systematic review?
Yes, it can be justified if the paper is on a specific technological development, policy, guideline, or subject. For example, limiting a search on developments in non-theoretical quantum computing may be only suitable from 1990 onwards, yet a search for the concept of loneliness should have no date limit. Any date limits should be mentioned and justified in your protocol/writeup.
Best Practice: Should I use a previously published literature search?
Yes. It can be considered best practice to use a published search strategies, however always consider a previously published search to be a framework to build upon, rather than to be used unmodified. Search strategies are tested and improved over time, and the quality can vary. The ISSG Search Filters Website includes examples of many different search strategies focusing on various study methodologies, and searching for (Your Subject) AND ("search stratg*") in the titles of papers in any database will provide validated search strategies for you to build upon. Systematic Review protocols are also valuable when developing your search, if you found a similar review during your scoping search.
During your literature search, you should consider search strategy testing by selecting 6-10 high-impact papers and checking that the papers are included in your search. These are often referred to as 'gold standard papers'.
How do I select my gold standard papers?
This is up to you, but your Faculty Librarian can advise. The papers should always be considered a 'strong include' in your systematic review, and can be found by reviewing references in other high quality systematic reviews, via a google scholar search, by identifying papers with high numbers of citations or simply by including well known, high impact papers.
How do I do I search strategy test?
Search your database with the title of the paper in quotation marks "Health literacy: a necessity for increasing participation in health care". You will usually only get one result.
Then combine this with your final search with AND and if the 1 remains, the paper is included in your search.
All my papers are included, what do I do now?
Congratulations. Looks like your search was excellent. You now have a tested highly sensitive search.
A few papers were not included in my search?
Review the abstract and keywords. You may have missed a search term. In qualitative research this is quite common as terminology is less strict, however introducing 'adjacent' or 'near field' searching can improve the search without increasing the number of search results too much.
Report on the strategy testing with a short line in the paper or protocol. "Search strategy testing was performed with a gold standard set of references identified by a (insert method here) and agreed with all members of the team"
Systematic Reviews are an 'evolving field' and one element that is growing in importance is transferring searches from complex fully featured databases such as MEDLINE or GREENFILE to interfaces with less functionality, and a much more basic search. This is more common in disciplines where the systematic review is nascent in adoption.
A few tips.
The new database does not have similar subject terms or none at all?
The solution is to add the subject terms as freetext. If possible use ADJ or N proximity searches combined with synonyms to improve the sensitivity.
DE "SOIL acidification" could become ((soil OR dirt OR earth OR ground OR clay OR loam OR topsoil OR turf OR humus) N5 acid*)
The database does not allow N or Proximity searching!
Add as free-text terms, and then perform a search strategy test to ensure you have captured the 'gold-standard' citations. It can get quite long!
((soil OR dirt OR earth OR ground OR clay OR loam OR topsoil OR turf OR humus) N5 acid*) could become ( ((soil OR dirt OR earth OR ground OR clay OR loam OR topsoil OR turf OR humus) acid*) OR (acid* (soil OR dirt OR earth OR ground OR clay OR loam OR topsoil OR turf OR humus)) OR ("acid* of" (soil OR dirt OR earth OR ground OR clay OR loam OR topsoil OR turf OR humus)) )
The database has an error 'Only 7 nested operands are allowed/search too complex' or just crashes
Systematic review searches can have issues with less developed interfaces. If you can break the search down into multiple 'mini searches' (the brackets are a good hint as to what can be broken down. Perform multiple searches then combine afterwards using reference management software and deduplication.
Downloadable List of Search Operators
Jason Curtis, from Shrewsbury and Telford Health Libraries has produced this excellent document, Search operators in bibliographic databases that lists all the variations between databases. Very useful when moving search strategies between databases.
The most important thing of all
Don't worry. The aim of a good search is to provide a high recall of relevant references in a single database. By combining multiple databases you improve the accuracy of the search and reduce the risk of missed papers.
The addition of multidisciplinary databases with weak functionality will always improve the search and combined with an effective search strategy test you should feel confident you have found most of the papers.
Consider your initial searching to be a delicious cake, and all the additional databases to be the cherries on that cake!
The paper Wanyama, S., McQuaid, R., & Kittler, M. (2022). Where you search determines what you find: The effects of bibliographic databases on systematic reviews is interesting additional reading if you are curious about the importance of multiple database searching, especially in non-health topics.
A PRISMA flow diagram is a visual representation of your search strategy depicting your database selection, the numbers of results found in each search, and excluded papers. It is recommended to produce one of these diagrams.
Templates for PRISMA Flow diagrams can be found at the PRISMA Website
What about other papers found outside of the search?
Often you might wish to add additional papers though 'other sources'. This is ok to do and in a Prisma flow diagram you can add 'additional resources searched'. As a recommendation try to keep this to a minimal amount as a criticism of a paper with a high number of 'other sources' could be that the literature search was insufficiently sensitive.
You can also add papers identified via Grey Literature searching or via Hand Searching of repositories not indexed in major databases (More common in worldwide topics). See the Grey Literature and Decolonising Literature Searching LibGuides for further information.
PRISMA-S
PRISMA 2020, an update of PRISMA also features PRISMA-S, an extension to PRISMA that focuses on the search. The paper PRISMA 2020 and PRISMA-S: common questions on tracking records and the flow diagram discusses some common PRISMA 2020 flow diagram questions.
It is highly recommended to make yourself aware of the papers How to Properly Use the PRISMA Statement and The PRISMA 2020 statement: an updated guideline for reporting systematic reviews as it is often reported incorrectly.
Deduplication is a key step for removing duplicate papers. A good literature search through multiple databases will result in significant duplication of references, so must be deduplicated to prevent wasted time during the screening step.
Endnote is recommended as it has the most robust deduplication features. The guidance video above references this paper https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4915647/ (Also read corrections) which features a multi step deduplication to save time and prevent the exclusion of suitable papers.
This is one method you can use for Screening, and other tools and methodologies exist. The paper High precision but variable recall – comparing the performance of five deduplication tools evaluates different screening tools other than Endnote.
Rayyan is a very useful web resource that allows you to screen papers quickly, and add reasons for exclusion so you can easily produce a PRISMA flow diagram. Once you have deduplicated your results in Endnote, export your references as a .RIS file into Rayyan for effective screening. (Guidance available in the FAQ article How do I export my references from Endnote into Mendeley or Rayyan? )
Once you have screened you can export from Rayyan back into Endnote and your search and screening are complete.
Access at Rayyan.ai. If you would like some assistance please contact a member of the faculty librarian team.
Rayyan is useful as it includes Keyboard Shortcuts to allow you to quickly screen papers:
The top 4 useful Keyboard Shortcuts
I = Include
U = Undecided
E = Exclude
R = Reason Drop Down Box (Type in box to add a new reason for exclusion)
Source of all the shortcuts Rayyan Keyboard Shortcuts
EndNote 20 is able to download full-text automatically, speeding up the review of screened papers. The How do I download full-text automatically with Endnote20 has guidance on setting up Endnote.
You are now onto the next most exciting part of your systematic review, either writing up your findings in the case of a narrative or rapid review, or producing your Meta-analysis.
If you are new to Statistics, or are looking for an introduction to Qualitative or Quantitative data tools such as SPSS or NVivo, LinkedIn Learning provides a range of instructional videos.
We wish you the very best, and further support is available in your department or from your supervisor.
Guidance on identifying additional evidence with a worldwide focus can be found in the Decolonising Literature Searching LibGuide