A systematic review (SR) – as the name implies – is a type of study or literature review that uses systematic methods to gather all existing evidence (from different sources), to answer specific questions. As the steps of a systematic review follow a certain methodology, they are replicable. That means if someone repeats the exact same steps as outlined in the review report, he or she should be able to obtain the same findings. Given that the data collected through SRs is derived from other (often primary) studies, SRs are generally classified as secondary sources. In the hierarchy of evidence, SRs and meta-analyses are at the top of the pyramid.
SRs aim at gathering the highest amount of evidence possible with regards to a specific research question, to enable appraisal, comparison and summarization of results. Or simply put, the purpose of SRs is to deliver a meticulous summary of all the available primary research. For instance, if a researcher wants to find out whether chronic psychological abuse causes metabolic syndrome, he/she might not be satisfied with the results of two or three studies within a particular geographical region. To ascertain the relationship, one should take into account all available studies (published and unpublished) from all possible sources. With all relevant studies, one can now have a more comprehensive view and come to a conclusion.
Systematic reviews are not easy to conduct. Often they need a minimum of three people, and sometimes more. The steps are lengthy and rigorous, but if conducted properly, they are of great value – especially in informing policies, promoting best practice in clinical management, and helping investigators to identify research gaps. There are many steps to conduct a systematic review and there are hundreds of articles out there describing them in details. This article however, aims at outlining the steps of SRs in a simplified manner.
- Planning your systematic review
It is important to plan your review process in great details. This is to avoid problems in the later stages. Things that should be planned include:
- How many people (researchers) are involved and what are the task descriptions for each
- What are the research questions
- What are the inclusion and exclusion criteria (what kind of study you want to include or exclude)
- Which database (and grey literature source) you will look into
- Methods of data extraction
- How findings should be presented
- Timeline (or you’ll never finish!)
- Starting the search
Generally the sources of SRs are electronic databases (where you can easily find published, peer-reviewed studies) and grey literature. Grey literature refers to all kinds of documents other than published studies in scientific journals. These comprise conference proceedings, government or NGO reports, relevant websites, unpublished studies, dissertations and many more. To begin the search, you need a set of keywords or search terms which have usually been planned and tested earlier. Prior testing gives you an idea of how sensitive or specific the terms are, in determining the outcomes (search results).
For grey literature, sometimes rigid keywords or search terms cannot be used, so the researcher must be more flexible and use his/her judgment as to how to search effectively. It is also good to consult field experts to find out all the possible sources that you can investigate, to avoid missing any.
- Title and abstract screening
Once you have all the results with you and you have removed duplicates, the next step is to screen through all the study titles and abstracts. Usually a minimum of two people are needed to accomplish this, and they should do the screening independently. At the end of the process, a comparison is made between the two, to see if similar studies are being included and excluded (consistency check). Differences are solved either by discussion (between the two reviewers) or consulting a third, independent person.
Sometimes, an abstract is not clear enough to indicate whether a study is relevant to the SR’s objective and criteria. In such a case, it should be considered potentially relevant and taken to the next step. Do not forget to document the numbers of titles and abstracts included or excluded, and the reason for exclusion, along the way.
- Full-text screening
All studies which have passed the initial phases will be retrieved and carefully read. One of the challenges at this stage is the inability to access full-texts. This can be due to papers not being available for free, journals not being subscribed by your institutions, papers simply available as abstracts (the actual study is not complete yet or never published), or papers being shared only upon requests or permission.
Some tips to help you obtain full-texts include: a) write to the corresponding author; b) write to the website to obtain permission to get the paper; c) check if the study is available in other sites such as academia or research gate, or drop a message to the author through these academic websites, and; d) ask a friend (another researcher) who has access to these journals, to get the paper for you (hey, what are friends for?)
- Quality appraisal
Sometimes there is a need to determine the quality of a paper (how good the paper is), so that you can judge the robustness of its findings. For this, you can either use existing quality appraisal tools such as the Newcastle-Ottawa Scale (NOS) and Cochrane Risk of Bias Tool, or you can modify the existing tools according to the requirements of your review.
- Data extraction
This is the process of taking out relevant information from a study and categorizing them accordingly. Extracting data requires a form – called the data extraction form – which is usually piloted beforehand (to ensure its feasibility and relevance). Sometimes, data extraction and full-text screening are done simultaneously. If two or more people are involved in extraction, it is helpful to compare extracted information from similar studies in order to maintain a degree of consistency. You can either create your own extraction form, or use existing ones from previous studies (which you can modify according to your needs).
- Data synthesis
This is the last step before writing your report or review. Data synthesis refers to the process of making sense of your gathered results – classifying, analyzing, and summarizing them in a systematic manner. This is crucial because it determines whether or not, and how your research questions will be answered. Conducting data synthesis properly also helps readers, users and stakeholders in understanding your overall review findings.
An SR is never perfect, but it represents an exhaustive search for a particular research question. You may come across obstacles such as: a) not being able to retrieve a number of papers (due to lack of response from the author or website); b) realizing that you missed some sources of grey literature; c) having to limit your review to a specific language or time period, and so on. What’s more important is that you acknowledge the limitations you have, and discuss them transparently while writing your report. This does not only reflect your integrity as a researcher, but it assists users (readers) in making the right judgment when reading your review.
Research and Publications Unit
IKRAM Health