- Awards Season
- Big Stories
- Pop Culture
- Video Games
- Celebrities
Where Can I Get Help Writing My Thesis Online?
You’ve spent years preparing for your master’s degree or PhD. You’ve read, studied and spent hours of time and energy writing papers. Now you’ve arrived at the culmination of all this effort: writing your thesis. There are plenty of compelling stories about the time and energy that students have spent drafting their dissertations and theses.
The good news is that you’re not alone. While you certainly don’t want to hire someone to write your thesis for you, which goes against most institution policies and puts your academic integrity at risk, you can get plenty of help with certain aspects of your thesis online. Whether you’re looking for a little guidance or extensive assistance, various services can make writing or editing your thesis go smoothly.
Dissertation Editor
One of the greatest challenges of writing your thesis can be juggling your family or job responsibilities with your studies. The time that writing takes can add another layer of obligation to your already-packed schedule. Dissertation Editor is a company whose founder is a PhD-educated writer and professor, and it promises to help you complete your thesis or dissertation on time and in compliance with your university’s rules and regulations.
Dissertation Editor’s primary function is to guide you along in the writing process and provide a helping hand in understanding everything you need to take care of. It places you with a writer who specializes in your area of study, and this individual can help you organize and analyze your research while making sure that your thesis fits your writing style and personality. This company also specializes in helping with any statistical analysis that you use in your thesis.
Thesis Helpers
If you’re concerned about using a service to help you write your thesis because you think it’ll be obvious that you hired help, don’t worry. Thesis Helpers puts its team of experienced writers to work for you to help you craft a thesis that finishes your degree on a high note. No matter what level of help you need, from narrowing down a topic to advanced editing and proofreading, they’re available to help.
The writers have advanced degrees in their areas of expertise, and one of the best things about Thesis Helpers is that it gives you ultimate say in the final product of your thesis. This company can help you with revisions and additional research, and you can rest assured that your thesis will meet anti-plagiarism standards.
Best Dissertation
Sometimes when you’re writing a thesis or dissertation, you can get stuck on one section or chapter. You may not need assistance writing the whole thing, but getting some help with the exact portion you’re struggling with can come in handy. That’s one of the strengths of using Best Dissertation . You don’t have to rely on it for help with your entire thesis if it’s not what you need.
Like most of the top thesis-assistance services, Best Dissertation employs writers with advanced degrees who specialize in various fields of study. What truly sets this company apart is the live support that it offers any time of the day or night. It claims to take the stress and strain out of writing your dissertation or thesis.
While some companies place a premium on helping you get your thesis written, others emphasize the editing and proofreading process. If you don’t need help with writing but need a hand with proofreading and editing, Scribbr is a good option for you. Its editors can help you get a grasp on the grammar and tone that are appropriate for academic writing.
Scribbr doesn’t just provide boilerplate feedback that you can find anywhere. It offers personalized feedback aimed at helping you become a better writer in the long run. You can even see examples of how its editors work by looking at the company’s website.
My Assignment Help
Writing a thesis has its own challenges that other academic writing simply doesn’t, which is why the team at My Assignment Help offers its particular brand of expertise. If you need assistance with a dissertation or thesis at the PhD or master’s level, its writers have the level of education and experience to help you write an expertly crafted and edited thesis.
My Assignment Help prides itself on hiring subject matter experts, meaning you can pair up with a helper who already has an advanced degree in your field. They understand the nuances of academic writing that are specific to your area of study, and they can provide advice on everything from making your abstract more unique to crafting a thought-provoking conclusion.
MORE FROM ASK.COM
Jan 10, 2022
Member-only
Can I conduct a Systematic Review as my Master’s dissertation or PhD thesis? Yes, It Depends!
First , when talking about the systematic review, I refer to it as a process, not a product . You may follow the systematic reviewing process, but the product may be called a scoping review , an umbrella review , a network meta-analysis , a realist review , an evidence gap and map , a systematic review and so on. So it does NOT mean that…
More from Farhad
A Proper Information Scientist/Professional with a Pinch of Career and Life Lessons
About Help Terms Privacy
Get the Medium app
Text to speech
- Open Access
- Published: 12 December 2017
Acceptance of a systematic review as a thesis: survey of biomedical doctoral programs in Europe
- Livia Puljak ORCID: orcid.org/0000-0002-8467-6061 1 , 2 , 3 &
- Damir Sapunar 3
Systematic Reviews volume 6 , Article number: 253 ( 2017 ) Cite this article
12k Accesses
20 Citations
79 Altmetric
Metrics details
Systematic reviews (SRs) have been proposed as a type of research methodology that should be acceptable for a graduate research thesis. The aim of this study was to analyse whether PhD theses in European biomedical graduate programs can be partly or entirely based on SRs.
In 2016, we surveyed individuals in charge of European PhD programs from 105 institutions. The survey asked about acceptance of SRs as the partial or entire basis for a PhD thesis, their attitude towards such a model for PhD theses, and their knowledge about SR methodology.
We received responses from 86 individuals running PhD programs in 68 institutions (institutional response rate of 65%). In 47% of the programs, SRs were an acceptable study design for a PhD thesis. However, only 20% of participants expressed a personal opinion that SRs meet the criteria for a PhD thesis. The most common reasons for not accepting SRs as the basis for PhD theses were that SRs are ‘not a result of a PhD candidate’s independent work, but more of a team effort’ and that SRs ‘do not produce enough new knowledge for a dissertation’. The majority of participants were not familiar with basic concepts related to SRs; questions about meta-analyses and the type of plots frequently used in SRs were correctly answered by only one third of the participants.
Conclusions
Raising awareness about the importance of SRs and their methodology could contribute to higher acceptance of SRs as a type of research that forms the basis of a PhD thesis.
Peer Review reports
Systematic reviews (SRs) are a type of secondary research, which refers to the analysis of data that have already been collected through primary research [ 1 ]. Even though SRs are a secondary type of research, a SR needs to start with a clearly defined research question and must follow rigorous research methodology, including definition of the study design a priori, data collection, appraisal of study quality, numerical analyses in the form of meta-analyses and other analyses when relevant and formulation of results and conclusions. Aveyard and Sharp defined SRs as ‘original empirical research’ because they ‘review, evaluate and synthesise all the available primary data, which can be either quantitative or qualitative’ [ 2 ]. Therefore, a SR represents a new research contribution to society and is considered the highest level in the hierarchy of evidence in medicine [ 3 ].
SRs have been proposed as a type of research methodology that should be acceptable as the basis for a graduate research thesis [ 4 , 5 ]. To the best of our knowledge, there are no reports on the acceptance of SRs as the basis for PhD theses. A recent review addressed potential advantages and disadvantages of such a thesis type and presented opposing arguments about the issue [ 5 ]. However, there were no actual data that would indicate how prevalent one opinion is over another with regard to the acceptance of a SR as the primary research methodology for a PhD thesis. The aim of this cross-sectional study was to assess whether a PhD thesis in European biomedical graduate programs can be partly or entirely based on a SR, as well as to explore the attitudes and knowledge of individuals in charge of PhD programs with regard to a thesis of this type.
Participants
The Organization of PhD Education in Biomedicine and Health Sciences in the European System (ORPHEUS) includes 105 institutional members from 40 countries and six associate members from Canada, Georgia, Iran, Kyrgyzstan, Kazakhstan and the USA [ 6 ]. The ORPHEUS encompasses a network of higher education institutions committed to developing and disseminating best practice within PhD training programs in biomedicine, health sciences and public health. ORPHEUS approved the use of their mailing list for the purpose of this study. The mailing list had 1049 contacts. The study authors were not given the mailing list due to data protection and privacy. Instead, it was agreed that ORPHEUS officials would send the survey via email to the mailing list. The General Secretary of the ORPHEUS contacted individuals responsible for PhD programs (directors or deputy directors) among the institutional members, via e-mail, on 5th of July 2016. These individuals were sent an invitation to complete an online survey about SRs as the basis for PhD theses. We invited only individuals responsible for PhD programs (e.g., directors, deputy directors, head of graduate school, vice deans for graduate school or similar). We also asked them to communicate with other individuals in charge of their program to make sure that only one person per PhD program filled out the survey. If there were several PhD programs within one institution, we asked for participation of one senior person per program.
The survey was administered via Survey Monkey (Portland, OR, USA). The survey took 5–10 min to complete. One reminder was sent to the targeted participants 1 month after the first mail.
The ethics committee of the University of Split School of Medicine approved this study, which formed part of the Croatian Science Foundation grant no. IP-2014-09-7672 ‘Professionalism in Health Care’.
Questionnaire
The 20-item questionnaire, designed specifically for this study by both authors (LP and DS), was first tested for face validity and clarity among five individuals in charge of PhD programs. The questionnaire was then modified according to their feedback. The questionnaire included questions about their PhD program; whether PhD candidates are required to publish manuscript(s) before thesis defence; the minimum number of required manuscripts for defending a PhD thesis; the authorship requirements for a PhD candidate with regard to published manuscript(s); whether there is a requirement for a PhD candidate to publish manuscript(s) in journals indexed in certain databases or journals of certain quality, and how the quality is defined; the description about other requirements for defending a PhD thesis; whether a SR partly or fully meets requirements for approval of a PhD thesis in their graduate program; what are the rules related to the use of a SR as the basis for a PhD thesis; and the number of PhD theses based on SRs relative to other types of research methods.
Participants were also asked about their opinion with regard to the main reasons that SRs are not recognised in some institutions as the basis for a doctoral dissertation, and their opinion about literature reviews, using a four-item Likert scale, ranging from ‘agree’ to ‘disagree’, including an option for ‘don’t know’. In the last question, the participants’ knowledge about SR methodology was examined using nine statements; participants had to rate each statement as either ‘correct’, ‘incorrect’, ‘unsure’ or ‘I don’t know’. Finally, participants were invited to leave their email address if they wanted to receive survey results. The survey sent to the study participants can be found in an additional file (Additional file 1 ).
Data analysis
Survey responses were entered into a spreadsheet, checked by both authors and analysed using Microsoft Excel (Microsoft Inc., Redmond, WA, USA). Descriptive data are presented as frequencies and percentages. All raw data and analysed data sets used in the manuscript are available from authors on request. A point-biserial correlation (SPSS, IBM, Chicago, IL, USA) was used to measure the strength of the association between results on the knowledge test (continuous variable) and the attitude towards SRs as the basis for dissertations (dichotomous variable; we used the answer to the following question as this measure: ‘Do you agree that a systematic review, in whole or in part, meets the criteria for a publication on which a doctoral dissertation can be based?’).
Study participants
There are 105 institutions included in the ORPHEUS network. We received a response from 86 individuals representing 68 institutions from 37 countries (65% institutional response rate). There were more respondents than institutions because some institutions have several PhD programs and thus several program directors. Those responders were used as a unit of analysis in the analysis of attitudes and knowledge; institutions were the unit of analysis when analysing criteria for theses. Some of the questionnaires ( n = 15) were only partly completed. In most cases, the missing data were related to knowledge about SR methodology.
Overview of requirements for a dissertation
Based on the information provided by the graduate program directors, in the majority of the included PhD programs, students were required to publish a research manuscript prepared within their PhD thesis prior to their thesis defence (83%; n = 64). Among 13 programs (17%) that did not have this requirement, five respondents (38%) indicated that in their opinion their school’s rules related to a PhD thesis should be changed such as to specify that each thesis should be based on work that is already published in a journal.
The minimum number of published manuscripts necessary for the PhD thesis defence was prespecified in 94% ( n = 60) of the programs that required publication of research manuscripts prior to the thesis defence. In most of the programs (37%; n = 22), the number of required manuscripts was three or more. Two manuscripts were required in 30% ( n = 18) and one was required in 33% ( n = 20) of the programs. In four programs, there was no formal policy on this matter, but there was a strong expectation that the student will have contributed substantially to several manuscripts in peer-reviewed journals.
In most cases, the PhD candidates’ contribution to published manuscripts within the PhD thesis was determined through first authorship. A requirement that a PhD candidate should be the first author on a manuscript(s) that constitutes a PhD thesis was reported in 82% ( n = 64) of the graduate programs.
In 60% ( n = 52) of the graduate programs, the quality of the journals where a PhD candidate has to publish research manuscripts as a part of a PhD thesis was defined by the database in which these journals are indexed. The most commonly specified databases were Web of Science (41%; n = 35) and MEDLINE/PubMed (13%; n = 11), followed by Science Citation Index, Scopus, Current Contents, a combination of several databases or, in two cases, a combination of journals from a list defined by some governing body.
Systematic reviews as a PhD thesis
SRs, in whole or in part, met the criteria for acceptable research methodology for a PhD thesis in 47% ( n = 40) of programs, whereas 53% ( n = 46) of programs specifically stated that they did not accept SRs in this context (Fig. 1 a, b). Among the programs that accepted SRs, theses could be exclusively based on a SR in 42% ( n = 17) of programs, while in the remaining programs, SRs were acceptable as one publication among others in a dissertation.
a European PhD programs that recognise a systematic review as a PhD thesis (green dot) and those that do not (red dot). Half red and half green dots indicate the five universities with institutions that have opposite rules regarding recognition of a systematic review as a PhD thesis. The pie chart presents b the percentage of the programs in which systematic reviews, in whole or in part, meet the criteria for a dissertation and c the opinion of participants about whether systematic reviews should form the basis of a publication within a PhD dissertation
The majority of participants (80%; n = 69) indicated that SRs did not meet criteria for a publication on which a PhD dissertation should be based (Fig. 1 c). The main arguments for not recognising a SR as the basis for a PhD thesis are listed in Table 1 . The majority of respondents were neutral regarding the idea that scoping reviews or SRs should replace traditional narrative reviews preceding the results of clinical and basic studies in doctoral theses. Most of the respondents agreed that narrative or critical/discursive literature reviews preceding clinical studies planned as part of a dissertation should be replaced with systematic reviews (Table 2 ).
Most of the programs that accepted SRs as a research methodology acceptable for PhD theses had defined rules related to the use of an SR as part of a PhD thesis (Fig. 2 ). The most common rule was that a SR can be one publication among others within a PhD thesis. Some of the respondents indicated that empty (reviews that did not find a single study that should be included after literature search) or updated reviews could also be used for a PhD thesis (Fig. 2 ).
Frequency of different rules that define the use of systematic reviews as a part of a PhD thesis in European biomedical graduate programs
The results of the survey regarding knowledge about SR methodology indicated that the majority of respondents were not familiar with this methodology. Only three out of nine questions were correctly answered by more than 80% of the participants, and questions about meta-analyses and the type of plots frequently used in a SR were correctly answered by only one third of the participants (Table 3 ). The association between participants’ results on the knowledge test and attitudes towards SRs was tested using a point-biserial correlation; this revealed that lack of knowledge was not correlated with negative attitudes towards SRs ( r pb = 0.011; P = 0.94).
In this study conducted among individuals in charge of biomedical graduate programs in Europe, we found that 47% of programs accepted SRs as research methodology that can partly or fully fulfil the criteria for a PhD thesis. However, most of the participants had negative attitudes about such a model for a PhD thesis, and most had insufficient knowledge about the basic aspects of SR methodology. These negative attitudes and lack of knowledge likely contribute to low acceptance of SRs as an acceptable study design to include in a PhD thesis.
A limitation of this study was that we relied on participants’ responses and not on assessments of formal rules of PhD programs. Due to a lack of familiarity with SRs, it is possible that the respondents gave incorrect answers. We believe that this might be the case since we received answers from different programs in the same university, where one person claimed that SRs were accepted in their program, and the other person claimed that they were not accepted in the other program. We had five such cases, so it is possible that institutions within the same university have different rules related to accepted research methodology in graduate PhD programs. This study may not be generalisable to different PhD programs worldwide that were not surveyed. The study is also not generalisable to Europe, as there are no universal criteria or expectations for PhD theses in Europe. Even in the same country, there may be different models and expectations for a PhD in different higher education institutions.
A recent study indicated a number of opposing views and disadvantages related to SRs as research methodology for graduate theses, including lack of knowledge and understanding by potential supervisors, which may prevent them from being mentors and assisting students to complete such a study [ 5 ]. This same manuscript emphasised that there may be constraints if the study is conducted in a resource-limited environment without access to electronic databases, that there may be a very high or very low number of relevant studies that can impact the review process, that methods may not be well developed for certain types of research syntheses and that it may be difficult to publish SRs [ 5 ].
Some individuals believe that a SR is not original research. Indeed, it has been suggested that SRs as ‘secondary research’ are different than ‘primary or original research’, implying that they are inferior and lacking in novelty and methodological rigour as compared to studies that are considered primary research. In 1995, Feinstein suggested that such studies are ‘statistical alchemy for the 21st century’ and that a meta-analysis removes or destructs ‘scientific requirements that have been so carefully developed and established during the 19th and 20th centuries’ [ 7 ]. There is little research about this methodological issue. Meerpohl et al. surveyed journal editors and asked whether they consider SRs to be original studies. The majority of the editors indicated that they do think that SRs are original scientific contributions (71%) and almost all journals (93%) published SRs. That study also highlighted that the definition of original research may be a grey area [ 8 ]. They argued that, in an ideal situation, ‘the research community would accept systematic reviews as a research category of its own, which is defined by methodological criteria, as is the case for other types of research’ [ 8 ]. Biondi-Zoccai et al. pointed out that the main criteria to judge a SR should be its novelty and usefulness, and not whether it is original/primary or secondary research [ 9 ].
In our study, 80% of the participants reported negative attitudes, and more than half of the respondents agreed with a statement that SRs are ‘not a result of the candidate’s independent work since systematic reviews tend to be conducted by a team’. This opinion is surprising since other types of research are also conducted within a team, and single authorship is very rare in publications that are published within a PhD thesis. On the contrary, the mean number of authors of research manuscripts is continuously increasing [ 10 ]. At the very least, the authors of manuscripts within a PhD will include the PhD candidate and a mentor, which is a team in and of itself. Therefore, it is unclear why somebody would consider it a problem that a SR is conducted within a team.
The second most commonly chosen argument against such a thesis was that SRs ‘do not produce enough new knowledge for a dissertation’. The volume of a SR largely depends on the number of included studies and the available data for numerical analyses. Therefore, it is unfair to label a SR as a priori lacking in new knowledge. There are SRs with tens or hundreds of included studies, and some of them not only include meta-analyses, but also network meta-analyses, which are highly sophisticated statistical methods. However, limiting SRs within a thesis only to those with meta-analysis would be unfair because sometimes meta-analysis is not justified due to clinical or statistical heterogeneity [ 11 ] and the presence or absence of a meta-analysis is not an indicator of the quality of a SR. Instead, there are relevant checklists for appraising methodological and reporting quality of a SR [ 12 , 13 ].
The third most commonly chosen argument against SRs within PhD theses was ‘lack of adequate training of candidates in methodology of systematic reviews’. This could refer to either insufficient formal training or insufficient mentoring. The graduate program and the mentor need to ensure that a PhD candidate receives sufficient knowledge to complete the proposed thesis topic. Successful mentoring in academic medicine requires not only commitment and interpersonal skills from both the mentor and mentee, but also a facilitating institutional environment [ 14 ]. This finding could be a result of a lack of capacity and knowledge for conducting SRs in the particular institutions where the survey was conducted, and not general opinion related to learning a research method when conducting a PhD study. Formal training in skills related to SRs and research synthesis methods [ 15 , 16 ], as well as establishing research collaborations with researchers experienced in this methodology, could alleviate this concern.
One third of the participants indicated a ‘lack of appreciation of systematic review methodology among faculty members’ as a reason against such a thesis model. This argument, as well as the prevalent negative attitude towards SRs as PhD theses, perhaps can be traced to a lack of knowledge about SR methodology; however, although the level of knowledge was quite low in our study, there was no statistically significant correlation between knowledge and negative attitudes. Of the nine questions about SR research methodology, only three questions were correctly answered by more than half of the participants. This could be a cause for concern because it has been argued that any health research should begin with a SR of the literature [ 17 ]. It has also been argued that the absence of SRs in the context of research training might severely hamper research trainees and may negatively impact the research conducted [ 18 ]. Thus, it has been recommended that SRs should be included ‘whenever appropriate, as a mandatory part of any PhD program or candidature’ [ 18 ].
It has recently been suggested that the overwhelming majority of investment in research represents an ‘avoidable waste’ [ 19 ]. Research that is not necessary harms both the public and patients, because funds are not invested where they are really necessary, and necessary research may not be conducted [ 17 ]. This is valid not only for clinical trials, but also for other types of animal and human experiments [ 20 ]. SRs can help improve the design of new experiments by relying on current evidence in the field and by helping to clarify which questions still need to be addressed. SRs can be instrumental in improving methodological quality of new experiments, providing evidence-based recommendations for research models, reducing avoidable waste, and enabling evidence-based translational research [ 20 ].
Four respondents from three institutions indicated that empty SRs are accepted as a PhD thesis. While it makes sense to include such a SR as a part of the thesis to indicate lack of evidence in a certain field, it is highly unlikely that an entire thesis can be based on an empty SR, without a single included study.
There are many advantages of a SR as a graduate thesis [ 4 , 5 ], especially as a research methodology suitable for low-resource settings. A PhD candidate can prepare a Cochrane SR as a part of the PhD thesis, yielding a high-impact publication [ 4 ]. Non-Cochrane SRs can also be published in high-impact journals. A PhD candidate involved in producing a SR within a PhD thesis goes through the same research process as those conducting primary research, from setting up a hypothesis and a research question, to development of a protocol, data collection, data analysis and appraisal, and formulation of conclusions. Graduate programs can set limits, such as the prevention of empty reviews and the recognition of updated reviews as valid for a PhD thesis, and engage experienced researchers as advisors and within thesis evaluation committees, to ensure that a candidate will conduct a high-quality SR [ 4 ]. Conducting a SR should not be mandatory, but candidates and mentors willing to produce such research within a graduate program should be allowed to do so.
Further studies in this field could provide better insight into attitudes related to SRs as graduate theses and explore interventions that can be used to change negative attitudes and improve knowledge of SRs among decision-makers in graduate education.
Raising awareness about the importance of SRs in biomedicine, the basic aspects of SR methodology and the status of SRs as original secondary research could contribute to greater acceptance of SRs as potential PhD theses. Our results can be used to create strategies that will enhance acceptance of SRs among graduate education program directors.
Gopalakrishnan S, Ganeshkumar P. Systematic reviews and meta-analysis: understanding the best evidence in primary healthcare. J. Fam. Med Prim Care. 2013;2(1):9–14.
Article CAS Google Scholar
Aveyard H, Sharp P. A beginner’s guide to evidence-based practice in health and social care. Glasgow: McGraw Open Press University; 2011.
Google Scholar
Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: synthesis of best evidence for clinical decisions. Ann Intern Med. 1997;126(5):376–80.
Article CAS PubMed Google Scholar
Puljak L, Sambunjak D. Cochrane systematic review as a PhD thesis: an alternative with numerous advantages. Biochemia Medica. 2010;20(3):319–2.
ten Ham-Baloyi W, Jordan P. Systematic review as a research method in post-graduate nursing education. Health SA Gesondheid. 2016;21:120–8.
Article Google Scholar
Organisation for PhD Education in Biomedicine and Health Sciences in the European System (ORPHEUS). Available at: http://www.orpheus-med.org/ .
Feinstein AR. Meta-analysis: statistical alchemy for the 21st century. J Clin Epidemiol. 1995;48(1):71–9.
Meerpohl JJ, Herrle F, Reinders S, Antes G, von Elm E. Scientific value of systematic reviews: survey of editors of core clinical journals. PLoS One. 2012;7(5):e35732.
Article CAS PubMed PubMed Central Google Scholar
Biondi-Zoccai G, Lotrionte M, Landoni G, Modena MG. The rough guide to systematic reviews and meta-analyses. HSR proc intensive care cardiovascular anesth. 2011;3(3):161–73.
CAS Google Scholar
Baethge C. Publish together or perish: the increasing number of authors per article in academic journals is the consequence of a changing scientific culture. Some researchers define authorship quite loosely. Dtsch. Arztebl. Int. 2008;105(20):380–3.
PubMed PubMed Central Google Scholar
Higgins JPT, Thompson SG. Quantifying heterogeneity in a meta-analysis. Statist Med. 2002;21:1539–58.
Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JP, Clarke M, Devereaux PJ, Kleijnen J, Moher D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009;6(7):e1000100.
Article PubMed PubMed Central Google Scholar
Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, Porter AC, Tugwell P, Moher D, Bouter LM. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7:10.
Sambunjak D, Straus SE, Marusic A. A systematic review of qualitative research on the meaning and characteristics of mentoring in academic medicine. J Gen Intern Med. 2010;25(1):72–8.
Article PubMed Google Scholar
Balajic K, Barac-Latas V, Drenjancevic I, Ostojic M, Fabijanic D, Puljak L. Influence of a vertical subject on research in biomedicine and activities of the Cochrane collaboration branch on medical students’ knowledge and attitudes toward evidence-based medicine. Croat Med J. 2012;53(4):367–73.
Marusic A, Sambunjak D, Jeroncic A, Malicki M, Marusic M. No health research without education for research—experience from an integrated course in undergraduate medical curriculum. Med Teach. 2013;35(7):609.
Mahtani KR. All health researchers should begin their training by preparing at least one systematic review. J R Soc Med. 2016;109(7):264–8.
Olsson C, Ringner A, Borglin G. Including systematic reviews in PhD programmes and candidatures in nursing - ‘Hobson’s choice’? Nurse Educ Pract. 2014;14(2):102–5.
Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet. 2009;374(9683):86–9.
de Vries RB, Wever KE, Avey MT, Stephens ML, Sena ES, Leenaars M. The usefulness of systematic reviews of animal experiments for the design of preclinical and clinical studies. ILAR J. 2014;55(3):427–37.
Download references
Acknowledgements
The authors thank the ORPHEUS secretariat for administering the survey and the study participants for taking time to participate in the survey. We are grateful to Prof. Ana Marušić for the critical reading of the manuscript.
This research was funded by the Croatian Science Foundation, grant no. IP-2014-09-7672 ‘Professionalism in Health Care’. The funder had no role in the design of this study or its execution and data interpretation.
Availability of data and materials
The datasets used and/or analysed for the current study are available from the corresponding author on reasonable request.
Author information
Authors and affiliations.
Cochrane Croatia, University of Split School of Medicine, Šoltanska 2, 21000, Split, Croatia
Livia Puljak
Department for Development, Research and Health Technology Assessment, Agency for Quality and Accreditation in Health Care and Social Welfare, Planinska 13, 10000, Zagreb, Croatia
Laboratory for Pain Research, University of Split School of Medicine, Šoltanska 2, 21000, Split, Croatia
Livia Puljak & Damir Sapunar
You can also search for this author in PubMed Google Scholar
Contributions
Both authors participated in the study design, data collection and analysis and writing of the manuscript, and both read and approved the final manuscript.
Corresponding author
Correspondence to Livia Puljak .
Ethics declarations
Ethics approval and consent to participate.
The Ethics Committee of the University of Split School of Medicine approved the study. All respondents consented to participate in the study.
Consent for publication
Not applicable.
Competing interests
The authors declare that they have no competing interests.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Additional file
Additional file 1:.
Online survey used in the study. Full online survey that was sent to the study participants. (PDF 293 kb)
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.
Reprints and Permissions
About this article
Cite this article.
Puljak, L., Sapunar, D. Acceptance of a systematic review as a thesis: survey of biomedical doctoral programs in Europe. Syst Rev 6 , 253 (2017). https://doi.org/10.1186/s13643-017-0653-x
Download citation
Received : 29 August 2017
Accepted : 30 November 2017
Published : 12 December 2017
DOI : https://doi.org/10.1186/s13643-017-0653-x
Share this article
Anyone you share the following link with will be able to read this content:
Sorry, a shareable link is not currently available for this article.
Provided by the Springer Nature SharedIt content-sharing initiative
- Systematic review
- PhD program
- Biomedicine
- Study design
Systematic Reviews
ISSN: 2046-4053
- Submission enquiries: Access here and click Contact Us
- General enquiries: [email protected]
- Open Access
- Published: 01 August 2019
A step by step guide for conducting a systematic review and meta-analysis with simulation data
- Gehad Mohamed Tawfik 1 , 2 ,
- Kadek Agus Surya Dila 2 , 3 ,
- Muawia Yousif Fadlelmola Mohamed 2 , 4 ,
- Dao Ngoc Hien Tam 2 , 5 ,
- Nguyen Dang Kien 2 , 6 ,
- Ali Mahmoud Ahmed 2 , 7 &
- Nguyen Tien Huy 8 , 9 , 10
Tropical Medicine and Health volume 47 , Article number: 46 ( 2019 ) Cite this article
544k Accesses
138 Citations
93 Altmetric
Metrics details
The massive abundance of studies relating to tropical medicine and health has increased strikingly over the last few decades. In the field of tropical medicine and health, a well-conducted systematic review and meta-analysis (SR/MA) is considered a feasible solution for keeping clinicians abreast of current evidence-based medicine. Understanding of SR/MA steps is of paramount importance for its conduction. It is not easy to be done as there are obstacles that could face the researcher. To solve those hindrances, this methodology study aimed to provide a step-by-step approach mainly for beginners and junior researchers, in the field of tropical medicine and other health care fields, on how to properly conduct a SR/MA, in which all the steps here depicts our experience and expertise combined with the already well-known and accepted international guidance.
We suggest that all steps of SR/MA should be done independently by 2–3 reviewers’ discussion, to ensure data quality and accuracy.
SR/MA steps include the development of research question, forming criteria, search strategy, searching databases, protocol registration, title, abstract, full-text screening, manual searching, extracting data, quality assessment, data checking, statistical analysis, double data checking, and manuscript writing.
Introduction
The amount of studies published in the biomedical literature, especially tropical medicine and health, has increased strikingly over the last few decades. This massive abundance of literature makes clinical medicine increasingly complex, and knowledge from various researches is often needed to inform a particular clinical decision. However, available studies are often heterogeneous with regard to their design, operational quality, and subjects under study and may handle the research question in a different way, which adds to the complexity of evidence and conclusion synthesis [ 1 ].
Systematic review and meta-analyses (SR/MAs) have a high level of evidence as represented by the evidence-based pyramid. Therefore, a well-conducted SR/MA is considered a feasible solution in keeping health clinicians ahead regarding contemporary evidence-based medicine.
Differing from a systematic review, unsystematic narrative review tends to be descriptive, in which the authors select frequently articles based on their point of view which leads to its poor quality. A systematic review, on the other hand, is defined as a review using a systematic method to summarize evidence on questions with a detailed and comprehensive plan of study. Furthermore, despite the increasing guidelines for effectively conducting a systematic review, we found that basic steps often start from framing question, then identifying relevant work which consists of criteria development and search for articles, appraise the quality of included studies, summarize the evidence, and interpret the results [ 2 , 3 ]. However, those simple steps are not easy to be reached in reality. There are many troubles that a researcher could be struggled with which has no detailed indication.
Conducting a SR/MA in tropical medicine and health may be difficult especially for young researchers; therefore, understanding of its essential steps is crucial. It is not easy to be done as there are obstacles that could face the researcher. To solve those hindrances, we recommend a flow diagram (Fig. 1 ) which illustrates a detailed and step-by-step the stages for SR/MA studies. This methodology study aimed to provide a step-by-step approach mainly for beginners and junior researchers, in the field of tropical medicine and other health care fields, on how to properly and succinctly conduct a SR/MA; all the steps here depicts our experience and expertise combined with the already well known and accepted international guidance.
Detailed flow diagram guideline for systematic review and meta-analysis steps. Note : Star icon refers to “2–3 reviewers screen independently”
Methods and results
Detailed steps for conducting any systematic review and meta-analysis.
We searched the methods reported in published SR/MA in tropical medicine and other healthcare fields besides the published guidelines like Cochrane guidelines {Higgins, 2011 #7} [ 4 ] to collect the best low-bias method for each step of SR/MA conduction steps. Furthermore, we used guidelines that we apply in studies for all SR/MA steps. We combined these methods in order to conclude and conduct a detailed flow diagram that shows the SR/MA steps how being conducted.
Any SR/MA must follow the widely accepted Preferred Reporting Items for Systematic Review and Meta-analysis statement (PRISMA checklist 2009) (Additional file 5 : Table S1) [ 5 ].
We proposed our methods according to a valid explanatory simulation example choosing the topic of “evaluating safety of Ebola vaccine,” as it is known that Ebola is a very rare tropical disease but fatal. All the explained methods feature the standards followed internationally, with our compiled experience in the conduct of SR beside it, which we think proved some validity. This is a SR under conduct by a couple of researchers teaming in a research group, moreover, as the outbreak of Ebola which took place (2013–2016) in Africa resulted in a significant mortality and morbidity. Furthermore, since there are many published and ongoing trials assessing the safety of Ebola vaccines, we thought this would provide a great opportunity to tackle this hotly debated issue. Moreover, Ebola started to fire again and new fatal outbreak appeared in the Democratic Republic of Congo since August 2018, which caused infection to more than 1000 people according to the World Health Organization, and 629 people have been killed till now. Hence, it is considered the second worst Ebola outbreak, after the first one in West Africa in 2014 , which infected more than 26,000 and killed about 11,300 people along outbreak course.
Research question and objectives
Like other study designs, the research question of SR/MA should be feasible, interesting, novel, ethical, and relevant. Therefore, a clear, logical, and well-defined research question should be formulated. Usually, two common tools are used: PICO or SPIDER. PICO (Population, Intervention, Comparison, Outcome) is used mostly in quantitative evidence synthesis. Authors demonstrated that PICO holds more sensitivity than the more specific SPIDER approach [ 6 ]. SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research type) was proposed as a method for qualitative and mixed methods search.
We here recommend a combined approach of using either one or both the SPIDER and PICO tools to retrieve a comprehensive search depending on time and resources limitations. When we apply this to our assumed research topic, being of qualitative nature, the use of SPIDER approach is more valid.
PICO is usually used for systematic review and meta-analysis of clinical trial study. For the observational study (without intervention or comparator), in many tropical and epidemiological questions, it is usually enough to use P (Patient) and O (outcome) only to formulate a research question. We must indicate clearly the population (P), then intervention (I) or exposure. Next, it is necessary to compare (C) the indicated intervention with other interventions, i.e., placebo. Finally, we need to clarify which are our relevant outcomes.
To facilitate comprehension, we choose the Ebola virus disease (EVD) as an example. Currently, the vaccine for EVD is being developed and under phase I, II, and III clinical trials; we want to know whether this vaccine is safe and can induce sufficient immunogenicity to the subjects.
An example of a research question for SR/MA based on PICO for this issue is as follows: How is the safety and immunogenicity of Ebola vaccine in human? (P: healthy subjects (human), I: vaccination, C: placebo, O: safety or adverse effects)
Preliminary research and idea validation
We recommend a preliminary search to identify relevant articles, ensure the validity of the proposed idea, avoid duplication of previously addressed questions, and assure that we have enough articles for conducting its analysis. Moreover, themes should focus on relevant and important health-care issues, consider global needs and values, reflect the current science, and be consistent with the adopted review methods. Gaining familiarity with a deep understanding of the study field through relevant videos and discussions is of paramount importance for better retrieval of results. If we ignore this step, our study could be canceled whenever we find out a similar study published before. This means we are wasting our time to deal with a problem that has been tackled for a long time.
To do this, we can start by doing a simple search in PubMed or Google Scholar with search terms Ebola AND vaccine. While doing this step, we identify a systematic review and meta-analysis of determinant factors influencing antibody response from vaccination of Ebola vaccine in non-human primate and human [ 7 ], which is a relevant paper to read to get a deeper insight and identify gaps for better formulation of our research question or purpose. We can still conduct systematic review and meta-analysis of Ebola vaccine because we evaluate safety as a different outcome and different population (only human).
Inclusion and exclusion criteria
Eligibility criteria are based on the PICO approach, study design, and date. Exclusion criteria mostly are unrelated, duplicated, unavailable full texts, or abstract-only papers. These exclusions should be stated in advance to refrain the researcher from bias. The inclusion criteria would be articles with the target patients, investigated interventions, or the comparison between two studied interventions. Briefly, it would be articles which contain information answering our research question. But the most important is that it should be clear and sufficient information, including positive or negative, to answer the question.
For the topic we have chosen, we can make inclusion criteria: (1) any clinical trial evaluating the safety of Ebola vaccine and (2) no restriction regarding country, patient age, race, gender, publication language, and date. Exclusion criteria are as follows: (1) study of Ebola vaccine in non-human subjects or in vitro studies; (2) study with data not reliably extracted, duplicate, or overlapping data; (3) abstract-only papers as preceding papers, conference, editorial, and author response theses and books; (4) articles without available full text available; and (5) case reports, case series, and systematic review studies. The PRISMA flow diagram template that is used in SR/MA studies can be found in Fig. 2 .
PRISMA flow diagram of studies’ screening and selection
Search strategy
A standard search strategy is used in PubMed, then later it is modified according to each specific database to get the best relevant results. The basic search strategy is built based on the research question formulation (i.e., PICO or PICOS). Search strategies are constructed to include free-text terms (e.g., in the title and abstract) and any appropriate subject indexing (e.g., MeSH) expected to retrieve eligible studies, with the help of an expert in the review topic field or an information specialist. Additionally, we advise not to use terms for the Outcomes as their inclusion might hinder the database being searched to retrieve eligible studies because the used outcome is not mentioned obviously in the articles.
The improvement of the search term is made while doing a trial search and looking for another relevant term within each concept from retrieved papers. To search for a clinical trial, we can use these descriptors in PubMed: “clinical trial”[Publication Type] OR “clinical trials as topic”[MeSH terms] OR “clinical trial”[All Fields]. After some rounds of trial and refinement of search term, we formulate the final search term for PubMed as follows: (ebola OR ebola virus OR ebola virus disease OR EVD) AND (vaccine OR vaccination OR vaccinated OR immunization) AND (“clinical trial”[Publication Type] OR “clinical trials as topic”[MeSH Terms] OR “clinical trial”[All Fields]). Because the study for this topic is limited, we do not include outcome term (safety and immunogenicity) in the search term to capture more studies.
Search databases, import all results to a library, and exporting to an excel sheet
According to the AMSTAR guidelines, at least two databases have to be searched in the SR/MA [ 8 ], but as you increase the number of searched databases, you get much yield and more accurate and comprehensive results. The ordering of the databases depends mostly on the review questions; being in a study of clinical trials, you will rely mostly on Cochrane, mRCTs, or International Clinical Trials Registry Platform (ICTRP). Here, we propose 12 databases (PubMed, Scopus, Web of Science, EMBASE, GHL, VHL, Cochrane, Google Scholar, Clinical trials.gov , mRCTs, POPLINE, and SIGLE), which help to cover almost all published articles in tropical medicine and other health-related fields. Among those databases, POPLINE focuses on reproductive health. Researchers should consider to choose relevant database according to the research topic. Some databases do not support the use of Boolean or quotation; otherwise, there are some databases that have special searching way. Therefore, we need to modify the initial search terms for each database to get appreciated results; therefore, manipulation guides for each online database searches are presented in Additional file 5 : Table S2. The detailed search strategy for each database is found in Additional file 5 : Table S3. The search term that we created in PubMed needs customization based on a specific characteristic of the database. An example for Google Scholar advanced search for our topic is as follows:
With all of the words: ebola virus
With at least one of the words: vaccine vaccination vaccinated immunization
Where my words occur: in the title of the article
With all of the words: EVD
Finally, all records are collected into one Endnote library in order to delete duplicates and then to it export into an excel sheet. Using remove duplicating function with two options is mandatory. All references which have (1) the same title and author, and published in the same year, and (2) the same title and author, and published in the same journal, would be deleted. References remaining after this step should be exported to an excel file with essential information for screening. These could be the authors’ names, publication year, journal, DOI, URL link, and abstract.
Protocol writing and registration
Protocol registration at an early stage guarantees transparency in the research process and protects from duplication problems. Besides, it is considered a documented proof of team plan of action, research question, eligibility criteria, intervention/exposure, quality assessment, and pre-analysis plan. It is recommended that researchers send it to the principal investigator (PI) to revise it, then upload it to registry sites. There are many registry sites available for SR/MA like those proposed by Cochrane and Campbell collaborations; however, we recommend registering the protocol into PROSPERO as it is easier. The layout of a protocol template, according to PROSPERO, can be found in Additional file 5 : File S1.
Title and abstract screening
Decisions to select retrieved articles for further assessment are based on eligibility criteria, to minimize the chance of including non-relevant articles. According to the Cochrane guidance, two reviewers are a must to do this step, but as for beginners and junior researchers, this might be tiresome; thus, we propose based on our experience that at least three reviewers should work independently to reduce the chance of error, particularly in teams with a large number of authors to add more scrutiny and ensure proper conduct. Mostly, the quality with three reviewers would be better than two, as two only would have different opinions from each other, so they cannot decide, while the third opinion is crucial. And here are some examples of systematic reviews which we conducted following the same strategy (by a different group of researchers in our research group) and published successfully, and they feature relevant ideas to tropical medicine and disease [ 9 , 10 , 11 ].
In this step, duplications will be removed manually whenever the reviewers find them out. When there is a doubt about an article decision, the team should be inclusive rather than exclusive, until the main leader or PI makes a decision after discussion and consensus. All excluded records should be given exclusion reasons.
Full text downloading and screening
Many search engines provide links for free to access full-text articles. In case not found, we can search in some research websites as ResearchGate, which offer an option of direct full-text request from authors. Additionally, exploring archives of wanted journals, or contacting PI to purchase it if available. Similarly, 2–3 reviewers work independently to decide about included full texts according to eligibility criteria, with reporting exclusion reasons of articles. In case any disagreement has occurred, the final decision has to be made by discussion.
Manual search
One has to exhaust all possibilities to reduce bias by performing an explicit hand-searching for retrieval of reports that may have been dropped from first search [ 12 ]. We apply five methods to make manual searching: searching references from included studies/reviews, contacting authors and experts, and looking at related articles/cited articles in PubMed and Google Scholar.
We describe here three consecutive methods to increase and refine the yield of manual searching: firstly, searching reference lists of included articles; secondly, performing what is known as citation tracking in which the reviewers track all the articles that cite each one of the included articles, and this might involve electronic searching of databases; and thirdly, similar to the citation tracking, we follow all “related to” or “similar” articles. Each of the abovementioned methods can be performed by 2–3 independent reviewers, and all the possible relevant article must undergo further scrutiny against the inclusion criteria, after following the same records yielded from electronic databases, i.e., title/abstract and full-text screening.
We propose an independent reviewing by assigning each member of the teams a “tag” and a distinct method, to compile all the results at the end for comparison of differences and discussion and to maximize the retrieval and minimize the bias. Similarly, the number of included articles has to be stated before addition to the overall included records.
Data extraction and quality assessment
This step entitles data collection from included full-texts in a structured extraction excel sheet, which is previously pilot-tested for extraction using some random studies. We recommend extracting both adjusted and non-adjusted data because it gives the most allowed confounding factor to be used in the analysis by pooling them later [ 13 ]. The process of extraction should be executed by 2–3 independent reviewers. Mostly, the sheet is classified into the study and patient characteristics, outcomes, and quality assessment (QA) tool.
Data presented in graphs should be extracted by software tools such as Web plot digitizer [ 14 ]. Most of the equations that can be used in extraction prior to analysis and estimation of standard deviation (SD) from other variables is found inside Additional file 5 : File S2 with their references as Hozo et al. [ 15 ], Xiang et al. [ 16 ], and Rijkom et al. [ 17 ]. A variety of tools are available for the QA, depending on the design: ROB-2 Cochrane tool for randomized controlled trials [ 18 ] which is presented as Additional file 1 : Figure S1 and Additional file 2 : Figure S2—from a previous published article data—[ 19 ], NIH tool for observational and cross-sectional studies [ 20 ], ROBINS-I tool for non-randomize trials [ 21 ], QUADAS-2 tool for diagnostic studies, QUIPS tool for prognostic studies, CARE tool for case reports, and ToxRtool for in vivo and in vitro studies. We recommend that 2–3 reviewers independently assess the quality of the studies and add to the data extraction form before the inclusion into the analysis to reduce the risk of bias. In the NIH tool for observational studies—cohort and cross-sectional—as in this EBOLA case, to evaluate the risk of bias, reviewers should rate each of the 14 items into dichotomous variables: yes, no, or not applicable. An overall score is calculated by adding all the items scores as yes equals one, while no and NA equals zero. A score will be given for every paper to classify them as poor, fair, or good conducted studies, where a score from 0–5 was considered poor, 6–9 as fair, and 10–14 as good.
In the EBOLA case example above, authors can extract the following information: name of authors, country of patients, year of publication, study design (case report, cohort study, or clinical trial or RCT), sample size, the infected point of time after EBOLA infection, follow-up interval after vaccination time, efficacy, safety, adverse effects after vaccinations, and QA sheet (Additional file 6 : Data S1).
Data checking
Due to the expected human error and bias, we recommend a data checking step, in which every included article is compared with its counterpart in an extraction sheet by evidence photos, to detect mistakes in data. We advise assigning articles to 2–3 independent reviewers, ideally not the ones who performed the extraction of those articles. When resources are limited, each reviewer is assigned a different article than the one he extracted in the previous stage.
Statistical analysis
Investigators use different methods for combining and summarizing findings of included studies. Before analysis, there is an important step called cleaning of data in the extraction sheet, where the analyst organizes extraction sheet data in a form that can be read by analytical software. The analysis consists of 2 types namely qualitative and quantitative analysis. Qualitative analysis mostly describes data in SR studies, while quantitative analysis consists of two main types: MA and network meta-analysis (NMA). Subgroup, sensitivity, cumulative analyses, and meta-regression are appropriate for testing whether the results are consistent or not and investigating the effect of certain confounders on the outcome and finding the best predictors. Publication bias should be assessed to investigate the presence of missing studies which can affect the summary.
To illustrate basic meta-analysis, we provide an imaginary data for the research question about Ebola vaccine safety (in terms of adverse events, 14 days after injection) and immunogenicity (Ebola virus antibodies rise in geometric mean titer, 6 months after injection). Assuming that from searching and data extraction, we decided to do an analysis to evaluate Ebola vaccine “A” safety and immunogenicity. Other Ebola vaccines were not meta-analyzed because of the limited number of studies (instead, it will be included for narrative review). The imaginary data for vaccine safety meta-analysis can be accessed in Additional file 7 : Data S2. To do the meta-analysis, we can use free software, such as RevMan [ 22 ] or R package meta [ 23 ]. In this example, we will use the R package meta. The tutorial of meta package can be accessed through “General Package for Meta-Analysis” tutorial pdf [ 23 ]. The R codes and its guidance for meta-analysis done can be found in Additional file 5 : File S3.
For the analysis, we assume that the study is heterogenous in nature; therefore, we choose a random effect model. We did an analysis on the safety of Ebola vaccine A. From the data table, we can see some adverse events occurring after intramuscular injection of vaccine A to the subject of the study. Suppose that we include six studies that fulfill our inclusion criteria. We can do a meta-analysis for each of the adverse events extracted from the studies, for example, arthralgia, from the results of random effect meta-analysis using the R meta package.
From the results shown in Additional file 3 : Figure S3, we can see that the odds ratio (OR) of arthralgia is 1.06 (0.79; 1.42), p value = 0.71, which means that there is no association between the intramuscular injection of Ebola vaccine A and arthralgia, as the OR is almost one, and besides, the P value is insignificant as it is > 0.05.
In the meta-analysis, we can also visualize the results in a forest plot. It is shown in Fig. 3 an example of a forest plot from the simulated analysis.
Random effect model forest plot for comparison of vaccine A versus placebo
From the forest plot, we can see six studies (A to F) and their respective OR (95% CI). The green box represents the effect size (in this case, OR) of each study. The bigger the box means the study weighted more (i.e., bigger sample size). The blue diamond shape represents the pooled OR of the six studies. We can see the blue diamond cross the vertical line OR = 1, which indicates no significance for the association as the diamond almost equalized in both sides. We can confirm this also from the 95% confidence interval that includes one and the p value > 0.05.
For heterogeneity, we see that I 2 = 0%, which means no heterogeneity is detected; the study is relatively homogenous (it is rare in the real study). To evaluate publication bias related to the meta-analysis of adverse events of arthralgia, we can use the metabias function from the R meta package (Additional file 4 : Figure S4) and visualization using a funnel plot. The results of publication bias are demonstrated in Fig. 4 . We see that the p value associated with this test is 0.74, indicating symmetry of the funnel plot. We can confirm it by looking at the funnel plot.
Publication bias funnel plot for comparison of vaccine A versus placebo
Looking at the funnel plot, the number of studies at the left and right side of the funnel plot is the same; therefore, the plot is symmetry, indicating no publication bias detected.
Sensitivity analysis is a procedure used to discover how different values of an independent variable will influence the significance of a particular dependent variable by removing one study from MA. If all included study p values are < 0.05, hence, removing any study will not change the significant association. It is only performed when there is a significant association, so if the p value of MA done is 0.7—more than one—the sensitivity analysis is not needed for this case study example. If there are 2 studies with p value > 0.05, removing any of the two studies will result in a loss of the significance.
Double data checking
For more assurance on the quality of results, the analyzed data should be rechecked from full-text data by evidence photos, to allow an obvious check for the PI of the study.
Manuscript writing, revision, and submission to a journal
Writing based on four scientific sections: introduction, methods, results, and discussion, mostly with a conclusion. Performing a characteristic table for study and patient characteristics is a mandatory step which can be found as a template in Additional file 5 : Table S3.
After finishing the manuscript writing, characteristics table, and PRISMA flow diagram, the team should send it to the PI to revise it well and reply to his comments and, finally, choose a suitable journal for the manuscript which fits with considerable impact factor and fitting field. We need to pay attention by reading the author guidelines of journals before submitting the manuscript.
The role of evidence-based medicine in biomedical research is rapidly growing. SR/MAs are also increasing in the medical literature. This paper has sought to provide a comprehensive approach to enable reviewers to produce high-quality SR/MAs. We hope that readers could gain general knowledge about how to conduct a SR/MA and have the confidence to perform one, although this kind of study requires complex steps compared to narrative reviews.
Having the basic steps for conduction of MA, there are many advanced steps that are applied for certain specific purposes. One of these steps is meta-regression which is performed to investigate the association of any confounder and the results of the MA. Furthermore, there are other types rather than the standard MA like NMA and MA. In NMA, we investigate the difference between several comparisons when there were not enough data to enable standard meta-analysis. It uses both direct and indirect comparisons to conclude what is the best between the competitors. On the other hand, mega MA or MA of patients tend to summarize the results of independent studies by using its individual subject data. As a more detailed analysis can be done, it is useful in conducting repeated measure analysis and time-to-event analysis. Moreover, it can perform analysis of variance and multiple regression analysis; however, it requires homogenous dataset and it is time-consuming in conduct [ 24 ].
Conclusions
Systematic review/meta-analysis steps include development of research question and its validation, forming criteria, search strategy, searching databases, importing all results to a library and exporting to an excel sheet, protocol writing and registration, title and abstract screening, full-text screening, manual searching, extracting data and assessing its quality, data checking, conducting statistical analysis, double data checking, manuscript writing, revising, and submitting to a journal.
Availability of data and materials
Not applicable.
Abbreviations
Network meta-analysis
Principal investigator
Population, Intervention, Comparison, Outcome
Preferred Reporting Items for Systematic Review and Meta-analysis statement
Quality assessment
Sample, Phenomenon of Interest, Design, Evaluation, Research type
Systematic review and meta-analyses
Bello A, Wiebe N, Garg A, Tonelli M. Evidence-based decision-making 2: systematic reviews and meta-analysis. Methods Mol Biol (Clifton, NJ). 2015;1281:397–416.
Article Google Scholar
Khan KS, Kunz R, Kleijnen J, Antes G. Five steps to conducting a systematic review. J R Soc Med. 2003;96(3):118–21.
Rys P, Wladysiuk M, Skrzekowska-Baran I, Malecki MT. Review articles, systematic reviews and meta-analyses: which can be trusted? Polskie Archiwum Medycyny Wewnetrznej. 2009;119(3):148–56.
PubMed Google Scholar
Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. 2011.
Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535.
Methley AM, Campbell S, Chew-Graham C, McNally R, Cheraghi-Sohi S. PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res. 2014;14:579.
Gross L, Lhomme E, Pasin C, Richert L, Thiebaut R. Ebola vaccine development: systematic review of pre-clinical and clinical studies, and meta-analysis of determinants of antibody response variability after vaccination. Int J Infect Dis. 2018;74:83–96.
Article CAS Google Scholar
Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, ... Henry DA. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008.
Giang HTN, Banno K, Minh LHN, Trinh LT, Loc LT, Eltobgy A, et al. Dengue hemophagocytic syndrome: a systematic review and meta-analysis on epidemiology, clinical signs, outcomes, and risk factors. Rev Med Virol. 2018;28(6):e2005.
Morra ME, Altibi AMA, Iqtadar S, Minh LHN, Elawady SS, Hallab A, et al. Definitions for warning signs and signs of severe dengue according to the WHO 2009 classification: systematic review of literature. Rev Med Virol. 2018;28(4):e1979.
Morra ME, Van Thanh L, Kamel MG, Ghazy AA, Altibi AMA, Dat LM, et al. Clinical outcomes of current medical approaches for Middle East respiratory syndrome: a systematic review and meta-analysis. Rev Med Virol. 2018;28(3):e1977.
Vassar M, Atakpo P, Kash MJ. Manual search approaches used by systematic reviewers in dermatology. Journal of the Medical Library Association: JMLA. 2016;104(4):302.
Naunheim MR, Remenschneider AK, Scangas GA, Bunting GW, Deschler DG. The effect of initial tracheoesophageal voice prosthesis size on postoperative complications and voice outcomes. Ann Otol Rhinol Laryngol. 2016;125(6):478–84.
Rohatgi AJaiWa. Web Plot Digitizer. ht tp. 2014;2.
Hozo SP, Djulbegovic B, Hozo I. Estimating the mean and variance from the median, range, and the size of a sample. BMC Med Res Methodol. 2005;5(1):13.
Wan X, Wang W, Liu J, Tong T. Estimating the sample mean and standard deviation from the sample size, median, range and/or interquartile range. BMC Med Res Methodol. 2014;14(1):135.
Van Rijkom HM, Truin GJ, Van’t Hof MA. A meta-analysis of clinical studies on the caries-inhibiting effect of fluoride gel treatment. Carries Res. 1998;32(2):83–92.
Higgins JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.
Tawfik GM, Tieu TM, Ghozy S, Makram OM, Samuel P, Abdelaal A, et al. Speech efficacy, safety and factors affecting lifetime of voice prostheses in patients with laryngeal cancer: a systematic review and network meta-analysis of randomized controlled trials. J Clin Oncol. 2018;36(15_suppl):e18031-e.
Wannemuehler TJ, Lobo BC, Johnson JD, Deig CR, Ting JY, Gregory RL. Vibratory stimulus reduces in vitro biofilm formation on tracheoesophageal voice prostheses. Laryngoscope. 2016;126(12):2752–7.
Sterne JAC, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355.
RevMan The Cochrane Collaboration %J Copenhagen TNCCTCC. Review Manager (RevMan). 5.0. 2008.
Schwarzer GJRn. meta: An R package for meta-analysis. 2007;7(3):40-45.
Google Scholar
Simms LLH. Meta-analysis versus mega-analysis: is there a difference? Oral budesonide for the maintenance of remission in Crohn’s disease: Faculty of Graduate Studies, University of Western Ontario; 1998.
Download references
Acknowledgements
This study was conducted (in part) at the Joint Usage/Research Center on Tropical Disease, Institute of Tropical Medicine, Nagasaki University, Japan.
Author information
Authors and affiliations.
Faculty of Medicine, Ain Shams University, Cairo, Egypt
Gehad Mohamed Tawfik
Online research Club http://www.onlineresearchclub.org/
Gehad Mohamed Tawfik, Kadek Agus Surya Dila, Muawia Yousif Fadlelmola Mohamed, Dao Ngoc Hien Tam, Nguyen Dang Kien & Ali Mahmoud Ahmed
Pratama Giri Emas Hospital, Singaraja-Amlapura street, Giri Emas village, Sawan subdistrict, Singaraja City, Buleleng, Bali, 81171, Indonesia
Kadek Agus Surya Dila
Faculty of Medicine, University of Khartoum, Khartoum, Sudan
Muawia Yousif Fadlelmola Mohamed
Nanogen Pharmaceutical Biotechnology Joint Stock Company, Ho Chi Minh City, Vietnam
Dao Ngoc Hien Tam
Department of Obstetrics and Gynecology, Thai Binh University of Medicine and Pharmacy, Thai Binh, Vietnam
Nguyen Dang Kien
Faculty of Medicine, Al-Azhar University, Cairo, Egypt
Ali Mahmoud Ahmed
Evidence Based Medicine Research Group & Faculty of Applied Sciences, Ton Duc Thang University, Ho Chi Minh City, 70000, Vietnam
Nguyen Tien Huy
Faculty of Applied Sciences, Ton Duc Thang University, Ho Chi Minh City, 70000, Vietnam
Department of Clinical Product Development, Institute of Tropical Medicine (NEKKEN), Leading Graduate School Program, and Graduate School of Biomedical Sciences, Nagasaki University, 1-12-4 Sakamoto, Nagasaki, 852-8523, Japan
You can also search for this author in PubMed Google Scholar
Contributions
NTH and GMT were responsible for the idea and its design. The figure was done by GMT. All authors contributed to the manuscript writing and approval of the final version.
Corresponding author
Correspondence to Nguyen Tien Huy .
Ethics declarations
Ethics approval and consent to participate, consent for publication, competing interests.
The authors declare that they have no competing interests.
Additional information
Publisher’s note.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Additional files
Additional file 1:.
Figure S1. Risk of bias assessment graph of included randomized controlled trials. (TIF 20 kb)
Additional file 2:
Figure S2. Risk of bias assessment summary. (TIF 69 kb)
Additional file 3:
Figure S3. Arthralgia results of random effect meta-analysis using R meta package. (TIF 20 kb)
Additional file 4:
Figure S4. Arthralgia linear regression test of funnel plot asymmetry using R meta package. (TIF 13 kb)
Additional file 5:
Table S1. PRISMA 2009 Checklist. Table S2. Manipulation guides for online database searches. Table S3. Detailed search strategy for twelve database searches. Table S4. Baseline characteristics of the patients in the included studies. File S1. PROSPERO protocol template file. File S2. Extraction equations that can be used prior to analysis to get missed variables. File S3. R codes and its guidance for meta-analysis done for comparison between EBOLA vaccine A and placebo. (DOCX 49 kb)
Additional file 6:
Data S1. Extraction and quality assessment data sheets for EBOLA case example. (XLSX 1368 kb)
Additional file 7:
Data S2. Imaginary data for EBOLA case example. (XLSX 10 kb)
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.
Reprints and Permissions
About this article
Cite this article.
Tawfik, G.M., Dila, K.A.S., Mohamed, M.Y.F. et al. A step by step guide for conducting a systematic review and meta-analysis with simulation data. Trop Med Health 47 , 46 (2019). https://doi.org/10.1186/s41182-019-0165-6
Download citation
Received : 30 January 2019
Accepted : 24 May 2019
Published : 01 August 2019
DOI : https://doi.org/10.1186/s41182-019-0165-6
Share this article
Anyone you share the following link with will be able to read this content:
Sorry, a shareable link is not currently available for this article.
Provided by the Springer Nature SharedIt content-sharing initiative
Tropical Medicine and Health
ISSN: 1349-4147
- Submission enquiries: Access here and click Contact Us
- General enquiries: [email protected]
Have a language expert improve your writing
Run a free plagiarism check in 10 minutes, generate accurate citations for free.
- Knowledge Base
- Methodology
- Systematic Review | Definition, Example, & Guide
Systematic Review | Definition, Example & Guide
Published on June 15, 2022 by Shaun Turney . Revised on December 7, 2022.
A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.
They answered the question “What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?”
In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.
Table of contents
What is a systematic review, systematic review vs. meta-analysis, systematic review vs. literature review, systematic review vs. scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, frequently asked questions about systematic reviews.
A review is an overview of the research that’s already been completed on a topic.
What makes a systematic review different from other types of reviews is that the research methods are designed to reduce bias . The methods are repeatable, and the approach is formal and systematic:
- Formulate a research question
- Develop a protocol
- Search for all relevant studies
- Apply the selection criteria
- Extract the data
- Synthesize the data
- Write and publish a report
Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.
Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.
Systematic reviews typically answer their research question by synthesizing all available evidence and evaluating the quality of the evidence. Synthesizing means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.
Systematic reviews often quantitatively synthesize the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.
A meta-analysis is a technique to synthesize results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .
Prevent plagiarism. Run a free check.
A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarize and evaluate previous work, without using a formal, explicit method.
Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.
Similar to a systematic review, a scoping review is a type of review that tries to minimize bias by using transparent and repeatable methods.
However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.
Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.
A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.
To conduct a systematic review, you’ll need the following:
- A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
- If you’re doing a systematic review on your own (e.g., for a research paper or thesis ), you should take appropriate measures to ensure the validity and reliability of your research.
- Access to databases and journal archives. Often, your educational institution provides you with access.
- Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
- Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.
A systematic review has many pros .
- They minimize research bias by considering all available evidence and evaluating each study for bias.
- Their methods are transparent , so they can be scrutinized by others.
- They’re thorough : they summarize all available evidence.
- They can be replicated and updated by others.
Systematic reviews also have a few cons .
- They’re time-consuming .
- They’re narrow in scope : they only answer the precise research question.
The 7 steps for conducting a systematic review are explained with an example.
Step 1: Formulate a research question
Formulating the research question is probably the most important step of a systematic review. A clear research question will:
- Allow you to more effectively communicate your research to other researchers and practitioners
- Guide your decisions as you plan and conduct your systematic review
A good research question for a systematic review has four components, which you can remember with the acronym PICO :
- Population(s) or problem(s)
- Intervention(s)
- Comparison(s)
You can rearrange these four components to write your research question:
- What is the effectiveness of I versus C for O in P ?
Sometimes, you may want to include a fifth component, the type of study design . In this case, the acronym is PICOT .
- Type of study design(s)
- The population of patients with eczema
- The intervention of probiotics
- In comparison to no treatment, placebo , or non-probiotic treatment
- The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
- Randomized control trials, a type of study design
Their research question was:
- What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?
Step 2: Develop a protocol
A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.
Your protocol should include the following components:
- Background information : Provide the context of the research question, including why it’s important.
- Research objective (s) : Rephrase your research question as an objective.
- Selection criteria: State how you’ll decide which studies to include or exclude from your review.
- Search strategy: Discuss your plan for finding studies.
- Analysis: Explain what information you’ll collect from the studies and how you’ll synthesize the data.
If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee . This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.
It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .
Step 3: Search for all relevant studies
Searching for relevant studies is the most time-consuming step of a systematic review.
To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:
- Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
- Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
- Gray literature: Gray literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of gray literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of gray literature.
- Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.
At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .
- Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
- Handsearch: Conference proceedings and reference lists of articles
- Gray literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
- Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics
Step 4: Apply the selection criteria
Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol . The third person’s job is to break any ties.
To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.
If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.
You should apply the selection criteria in two phases:
- Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
- Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.
It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarize what you did using a PRISMA flow diagram .
Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.
When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.
Step 5: Extract the data
Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:
- Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
- Your judgment of the quality of the evidence, including risk of bias .
You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group .
Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.
They also collected data about possible sources of bias, such as how the study participants were randomized into the control and treatment groups.
Step 6: Synthesize the data
Synthesizing the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesizing the data:
- Narrative ( qualitative ): Summarize the information in words. You’ll need to discuss the studies and assess their overall quality.
- Quantitative : Use statistical methods to summarize and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.
Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.
Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analyzed the effect sizes within each group.
Step 7: Write and publish a report
The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.
Your article should include the following sections:
- Abstract : A summary of the review
- Introduction : Including the rationale and objectives
- Methods : Including the selection criteria, search method, data extraction method, and synthesis method
- Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
- Discussion : Including interpretation of the results and limitations of the review
- Conclusion : The answer to your research question and implications for practice, policy, or research
To verify that your report includes everything it needs, you can use the PRISMA checklist .
Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.
A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .
It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.
A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other academic texts , with an introduction , a main body, and a conclusion .
An annotated bibliography is a list of source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a paper .
A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.
Cite this Scribbr article
If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.
Turney, S. (2022, December 07). Systematic Review | Definition, Example & Guide. Scribbr. Retrieved February 27, 2023, from https://www.scribbr.com/methodology/systematic-review/
Is this article helpful?
Shaun Turney
Other students also liked, how to write a literature review | guide, examples, & templates, how to write a research proposal | examples & templates, what is critical thinking | definition & examples, what is your plagiarism score.
Improve your thesis with a systematic literature review
What is a systematic literature review.
Systematic literature reviews are a means to rigorously review existing literature on a specific topic. They collect and analyse existing literature in a systematic and replicable way.
While systematic literature reviews require a lot of work, they can convincingly draw conclusions on the state of the art of existing knowledge, uncover research gaps, and support the creation of new theoretical and conceptual frameworks.
The difference between a systematic and a regular literature review
Before starting to consider whether a systematic literature review is right for you, it is important to be aware of the main differences between systematic and regular literature reviews.
The main differences between a systematic and a regular literature review are the process of collecting and analysing literature, the accuracy of claims, the scope and replicability.
Additionally, the scope of a review differs immensely between systematic and regular literature reviews.
T he difference between a systematic literature review and a meta-analysis
Benefits of conducting a systematic literature review.
There is of course nothing wrong with conducting a ‘regular’ literature review. However, there are several good reasons to conduct a systematic literature review.
A key benefit of conducting a systematic literature review is the knowledge and confidence that all relevant literature has been analysed within certain parameters, reducing the fear that one may have missed something.
Academic citation databases suitable for systematic literature reviews
Steps for conducting a systematic literature review, get new content delivered directly to your inbox, getting the most out of thesis supervision meetings, separating your self-worth from your phd work, related articles, 75 linking words for academic writing (+examples), how to deal with procrastination productively during thesis writing, sample emails to your thesis supervisor.
Library subject guides
Systematic reviews.
- Starting the review
- About systematic reviews
- Research Question
- Plan your search
- Sources to search
- Search example
- Screen and Analyse
What to include
Introduction, discussion and conclusion, annotated example.
- Guides and Software
- Help & Support
Suggested books
To help with the writing process try some books from the Library on:
- academic writing
- scholarly publishing
- literature review
- thesis writing
In general, the writing process for a systematic review is similar to the process of writing any other kind of review.
A systematic review should provide an answer to the research question , it is not a broad overview of current trends and gaps in research. The review should show the reader how the answer was found, and provide the results you have identified.
A systematic review must have a detailed methodology that describes the search process and the selection process. This is why careful documentation of the methodology is important. A reader of the review should be able to critically interpret the findings- to understand why sources were chosen, how they were assessed, and how conclusions were reached.
The structure of the systematic review differs from the narrative review or the traditional literature review that allows you to organise it to best support your argument. A systematic review should reflect the stages outlined in the protocol . With a systematic review reporting guidelines should be followed that help you identify what should be included in each section of the review. One such standard approach is PRISMA .
Although much time is invested in developing a search strategy and screening results, a systematic review is valued by the critical reflection and interpretation of the findings . Focus on analysing, not summarising. Use a critical analysis tool to assess the studies.
Your systematic review needs to tell a story, and it needs to clearly articulate how it provides meaningful and original advancement of the field .
The abstract provides an overview of the systematic review. It usually covers the following:
- A brief background (what we know and often the gap that the review will fill)
- The aim or hypothesis
- Summary of methods
- Summary of results
- Summary of conclusion (and sometimes recommendations).
Note that these points represent the general ‘story line’ seen in most systematic reviews: What we know (and perhaps what the gap is); what we set out to do; what we did; what we found; what this means.
The introduction provides an overview of the systematic review and enough contextual information for the reader to make sense of the remainder of the report. It usually covers the following:
- Background information to contextualise the review (what we already know about this area)
- Definitions of key terms and concepts if needed
- The rationale for the study (often in terms of a gap in knowledge that needs to be filled, a lack of agreement within the literature that needs to be resolved, or the potential implications of the findings)
- The aims and/or objectives (optional)
- The research question/s emanating from the rationale
- Additional information (Optional)
Note however, that these points are not always in this order. Some writers prefer to begin with the research questions, followed by the context, building to the rationale.
The methods section can be divided up into two main sections.
The first section describes how the literature search was conducted. This section may contain any of the following information:
- The databases searched and whether any manual searches were completed
- How search terms were identified
- What terms were employed in the key word searches
- If particular sections of articles were looked at during the search and collection stage i.e. titles, abstracts, table of contents ( note : the information in these sections may have informed the selection process)
The second section discusses the criteria for including or excluding studies. This section may include any of the following information:
- Your selection criteria
- How you identified relevant studies for further analysis
- What articles you reviewed
- What particular areas you looked at in the selected articles i.e. a relationship or association between two things (such as a genetic predisposition and a drug), the outcome measures of a health campaign, drug treatment, or clinical intervention, the differing impact of a particular drug or treatment.
Details about the kind of systematic review undertaken, i.e. thematic analysis, might also be mentioned in the methods section.
Broadly speaking, in the results section, everything you have done so far needs to be presented. This can include any of the following:
- briefly mention the databases used for the searching
- identify the number of hits
- show how the articles were selected by title, abstract, table of contents or other procedures.
- Overview of the kinds of studies selected for the review i.e. the types of methodologies or study designs used.
- where the trials were conducted
- treatment duration
- details about participants
- similarities and differences in the way data was measured
- similar or different approaches to the same treatment or condition
- Risk of bias across studies
- the kinds of relationships or associations demonstrated by the studies
- frequency of positive or adverse effects of a particular treatment or drug
- the number of studies that found a positive correlation between two phenomena or found a causal relationship between two variables
Often, researchers will include tables in the Results section or Appendix to provide on overview of data found in the studies. Remember, tables in the Results section need to be explained fully.
A primary function of your discussion and conclusion is to help readers understand the main findings and implications of the review.
The following elements are commonly found in the discussion and conclusion sections. Note that the points listed are neither mandatory nor in any prescribed order.
Discussion:
- Summary of main findings
- Interpretation of main findings (don’t repeat results)
- Strengths and weaknesses
- Comparison with previous review findings or general literature
- The degree to which the review answers the research question
- Whether the hypothesis was confirmed
- Limitations (e.g. biases, lack of methodological rigour or weak evidence in the articles)
Conclusion:
- Summary of how it answers the research question (the ‘take home’ message)
- Significance of the findings
- Reminder of the limitations
- Implications and recommendations for further research.
Separate or combined?
A key difference between a discussion and a conclusion relates to how specific or general the observations are. A discussion closely interprets results in the context of the review. A conclusion identifies the significance and the implications beyond the review. Some reviews present these as separately headed sections. Many reviews, however, present only one section using a combination of elements. This section may be headed either Discussion or Conclusion.
The following publication has been used to supply annotated examples of the abstract, introduction, discussion and conclusion.
Jørgensen, C. R., Thomsen, T. G., Ross, L., Dietz, S. M., Therkildsen, S., Groenvold, M., Rasmussen, C. L., & Johnsen, A. T. (2018). What facilitates “patient empowerment” in cancer patients during follow-up: A qualitative systematic review of the literature. Qualitative Health Research, 28(2), 292-304. https://doi.org/10.1177/1049732317721477
- Abstract from journal article - annotated example
- Introduction from journal article - annotated example
- Discussion and Conclusion from journal article - annotated example
- << Previous: Synthesise
- Next: Guides and Software >>
- Last Updated: Feb 27, 2023 10:54 AM
- URL: https://rmit.libguides.com/systematicreviews
- Study and research support
- Literature searching
Systematic reviews
What is a systematic review.
A systematic review is a complex piece of research that aims to identify, select and synthesise all research published on a particular question or topic.
Systematic reviews adhere to a strict scientific design based on pre-specified and reproducible methods. They provide reliable estimates about the effects of interventions.
As well as illustrating knowledge about a particular intervention, systematic reviews can also show where knowledge is lacking. You can use this to guide future research.
Usually, a systematic review will include a search methodology, in which you document where, when and how you looked for information, as well as who you consulted.
To find out more about the process of a systematic review, you may find Oxford University’s in-depth Systematic Reviews guide useful.
If you want to see examples of successful systematic reviews, you can search for them in various places such as on Cochrane library . Our search information on clinical trials and systematic reviews provides more information.
- Research Guides
- Health Science Libraries
Systematic Reviews & Other Review Types
What is a systematic review.
- Systematic Review Service
- Databases & Grey Literature
- Tools for Critical Appraisal
- What is a Rapid Review?
- What is a Scoping Review?
- What is a Mapping Review?
- What is a Meta-Synthesis?
- What is a Mixed Methods Review?
- What is an Integrative Review?
- What is an Overview of Reviews?
- What is a Review of Complex Interventions?
- What is a Diagnostic Test Accuracy Review?
- What is a Network Meta-Analysis?
- What is a Living Systematic Review?
Systematic Review vs. Literature (Narrative) Review
Traditional literature review / narrative review:
Characteristics:
- Describes and appraises previous work but does not describe specific methods by which the reviewed studies were identified, selected and evaluated
- Overviews, discussions, critiques of previous work and the current gaps in knowledge
- Often used as rationale for new research
- To scope the types of interventions available to include in a review
Limitations:
- The writers assumptions and agenda often unknown
- Biases that occur in selecting and assessing the literature are unknown
- Cannot be replicated
Systematic review:
- The scope of the review is identified in advance (eg review question and sub‐questions and/or sub‐group analysis to be undertaken)
- Comprehensive search to find all relevant studies
- Use of explicit criteria to include / exclude studies
- Application of established standards to critically appraise study quality
- Explicit methods of extracting and synthesizing study findings (qualitative or quantitative)
- May include a meta-analysis (quantitative synthesis) *optional
- Identifies, appraises and synthesizes all available research that is relevant to a particular review question
- Collates all that is known on a given topic and identifies the basis of that knowledge
- Comprehensive report using explicit processes so that rationale, assumptions and methods are open to scrutiny by external parties
- Can be replicated / updated
- Systematic reviews with narrowly defined review questions provide specific answers to specific questions
- Alternative questions that have not been answered usually need to be reconstructed by the reader
Source: Cochrane. Background to Systematic Reviews
This link will open a PDF document.
https://ph.cochrane.org/sites/ph.cochrane.org/
files/public/uploads/Unit_One.pdf
A systematic review is defined as “a review of the evidence on a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant primary research, and to extract and analyze data from the studies that are included in the review.” The methods used must be reproducible and transparent.
Source: Undertaking Systematic Reviews of Research on Effectiveness. CRD’s Guidance for those Carrying Out or Commissioning Reviews. CRD Report Number 4 (2nd Edition). NHS Centre for Reviews and Dissemination, University of York. March 2001.
Image: EBM Pyramid and EBM Page Generator, copyright 2006 Trustees of Dartmouth College and Yale University. All Rights Reserved. Produced by Jan Glover, David Izzo, Karen Odato and Lei Wang.
When is a Systematic Review the most appropriate study design?
When answering questions of effectiveness comparing two different treatments or interventions.
Is your review question a complex intervention? Learn more about Reviews of Complex Interventions .
Choosing a Review Type: This guide explains other comprehensive literature reviews of similar methodology to the systematic review.
Here is a helpful article about review types. ( Meeting the Review Family: Exploring review types and associated information retrieval requirements, 2019,Sutton et al.)
You may also find the Review Ready Reckoner helpful!
- What Review is Right for You -tool
- What Type of Review is Right for You?- Decision Tree
- Review Ready Reckoner - Assessment Tool (RRRsAT)1 Table of Review Types
7 Stages of Conducting a Systematic Review
1. Gathering your team (Minimum of two reviewers with a third to serve as a tiebreaker)
A systematic review must have a team of two or greater. A systematic review cannot be completed by one person. Choose team members wisely and based on areas of expertise. A third team member is sometimes called a tiebreaker. They are to resolve disagreements for reviewers 1 and 2 for stages of the review that are blinded (screening, data extraction, critical appraisal) and are completed by two independent reviewers.
2. Questioning (Define a narrow question, may use PICO)
The PICO format is commonly used to define the research question into one that is a searchable question. In some cases, the PICO format may not work and another format can be used. The P in PICO is Patient/Problem or Person. I is for Intervention/Exposure/Therapy or Treatment. C is optional and is for Comparison (such as a placebo or another drug/therapy) and O is for Outcome(s), what is the expected or anticipated outcome you will find in the literature? A systematic review question should be narrow in scope. The purpose of a systematic review is to draw conclusions based on the evidence to answer that one well-defined and narrow question.
3. Planning (Create a protocol, plan methods & strategies, register protocol) * This course focuses on the planning stage
Having a plan in place is essential to a good quality review and by spending more time planning before the review takes place, you could avoid issues or errors that may slow down the process or be detrimental to the review. Planning includes seeing if the review is feasible, checking to make sure there are no conflicting reviews and also ensuring that there is a plan to carry out each stage of the review. Setting goals and timelines for the review is important as well as mapping out how the review project will be managed. This is also put into a document called a protocol. Registering the protocol is optional but highly recommended. The protocol also includes defining a priori what the selection criteria will be for the review in terms of inclusion and exclusion criteria for what studies should be screened by for inclusion in the review.
4. Searching/Screening (Exhaustive, transparent & repeatable searching for evidence/selecting studies)
Includes searching multiple databases, grey literature/clinical trial registries and handsearching of the literature (performed by the subject matter expert). It is best practice to involve a librarian or an information specialist in creating the comprehensive search, translating the search for databases or grey literature, documenting the search and deduplicating the repeating references in a citation manager and writing the search methods for the review. However, librarians are usually not involved in grey literature searching unless they are an expert in the subject matter. The review team member with the most subject matter expertise is the one who is best equipped to handsearch. The search stage may also include contacting other experts in the field to identify publications that have not been published yet. Systematic reviews include both published and unpublished literature to avoid a type of publication bias, called positive outcomes bias since positive outcomes are more likely to be published. Screening is done in two phases. The first phase is screening titles/abstracts (together) and the second phase is screening full texts. Screening is done independently by two reviewers, with a third reviewer serving as a tiebreaker. Reviewers should not move on to the full text screening phase until they have screened all of the titles and abstracts and each is a clear Yes or No without maybes remaining. Once they are ready to screen full texts, they must acquire and read all of the full texts and screen them based on the studies selection criteria. Only Yes's are included in the review but all No's must have a reason listed for exclusion. The new PRISMA 2020 requires reporting of study Near Misses too. Near misses are any studies that did not meet inclusion in the review but were very close to being included. Refer to the PRISMA 2020 http://www.prisma-statement.org/PRISMAStatement/ for more guidance on this stage. There are tools designed specifically to assist with the systematic review screening phase.
5. Managing & reporting
All methods must be fully reported, transparent and reproducible. The methods reported must also follow the recommended reporting guidance such as the PRISMA 2020. Reporting guidance can be identified by searching the Equator Network https://www.equator-network.org/ . Reporting guidance may be modified for review types similar to the systematic review. Refer to the many PRISMA 2020 extensions http://www.prisma-statement.org/Extensions/ for more information.
6. Data Extraction/Synthesizing the evidence
This stage includes appraising the evidence, interpreting results, performing a qualitative (narrative analysis) and/or a quantitative/meta-analysis. A meta-analysis is optional and is only done if it is feasible. A biostatistician or advanced training in statistics is recommended if doing a meta-analysis. There are many tools designed to assist with this process.
Evidence from studies is assessed using critical appraisal or Risk of Bias tools and/or checklists by study design.
Critical appraisal tools from Temple University, Systematic Review Research Libguide
More tools for critical appraisal and other stages of the review from the National Collaborating Centre for Methods and Tools
Data from all studies must also be extracted and put into tables/charts such as the Summary of Findings (SOF) table and is reported as a narrative synthesis. Data is collected from all studies if conducting a meta-analysis and its numerical findings are reported.
Here are some more detailed elaborations and examples:
Synthesis: Provide a narrative synthesis of the included studies individually and when combined (What are the differences and the commonality between studies?) or what can be demonstrated from the research when combining the studies together? A meta-analysis is optional. Create a data abstraction/extraction form for the purposes of collecting data that is similar across all included studies, include a ‘Characteristics of Studies’ table to show this data (see table example and this example (opens a PDF document) on page 48. Summary of Findings tables are provided starting on page 8 of the same document. Data extraction must be done using data extraction forms and independently/blinded by two reviewers, with a 3rd reviewer serving as a tiebreaker.
7. Drawing Conclusions, Writing & Publishing
After completing these steps, the results of the review must be shared. What is the level of evidence? Is there evidence in support of the question or are more studies needed to draw conclusions? What are your recommendations for future studies? What are the limitations to your systematic review? How do these findings from your review change what is known on the topic or question?
Conclusions/Recommendations:
Discuss what contribution this review makes and how your review answers or addresses the original question. Discuss any gaps found in the research. Make recommendations for needed research to address these gaps and the importance of addressing them. Discuss the overall strength of evidence in support of your original question (strong, moderate or weak).
For more guidance on the systematic review stages, refer to the Cochrane Handbook (medicine/health sciences), the JBI Manual (health sciences/nursing) or the methods guides for Campbell Collaboration Systematic Reviews (Business, education, social welfare, criminal & justice topics and more).
Where to register your Systematic Review Protocol?
- PROSPERO -registry for medical/health systematic reviews
- OSF Preregistration Register a systematic review protocol, including non-medical/health topics. Choose to affiliate with Temple University.
- TUScholarShare-Temple IR
- SYREAF-systematic reviews for animals and food
- Joanna Briggs Institute Systematic Review Register
- Inclusive Systematic Review Registration Form This form, therefore, is a fall-back for more specialized forms and can be used if no specialized form or registration platform is available. Can be used for a variety of disciplines and review types.
Moller AM, Myles PS. What makes a good systematic review and meta-analysis? BJA. 2016. 117(4):428-430.
- How to do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses
- JBI Manual: Systematic Reviews of Effectiveness
- JBI Manual: Systematic Reviews of Prevalence and Incidence
- JBI Manual: Systematic Reviews for Etiology and Risk
- JBI Manual: Systematic Reviews of Measurement Properties
- JBI manual: Systematic Reviews of Text and Opinion
- How to review the evidence: Systematic identification and review of the scientific literature
- BMJ OPEN Systematic Reviews Systematic Review articles published in the BMJ Open (Open-Access) medical journal.
- Lockwood, Geum. Systematic reviews: Guidelines, tools and checklists for authors (2017)
Systematic Review Standards
- Finding What Works in Healthcare: Standards for Systematic Reviews National Academies of Science, Engineering, & Medicine (Formerly Institute of Medicine (IOM)) Standards for systematic reviews.
- Equator Reporting Guidelines Decision Tree A tool to select the appropriate reporting guideline for your review. This link will open a PDF document.
- PRISMA: Transparent Reporting of Reviews and Meta-Analyses Reporting guidelines for reviews of Interventions.
- PRISMA Checklist
- PRISMA Flow Diagram
- PRISMA-S extension for search reporting
- MOOSE (Meta-analysis of Observational Studies in Epidemiology) Guidelines Reporting guidelines for observational reviews of cohort, case-control or cross-sectional studies. This link will open a PDF document.
- Cochrane Handbook of Systematic Reviews for Interventions A guide for Cochrane reviewers, explains the Cochrane systematic review process in great detail.
- MECIR-Updated Cochrane Methodological Expectations of Cochrane Intervention Reviews
- JBI Manual for Evidence Synthesis
- Campbell Collaboration
- The Campbell Collaboration MECCIR standards These standards are adapted from the Cochrane MECIR standards and are for disciplines beyond medicine such as crime, justice, education, international development, methods, social welfare, disability, business knowledge translation/implementation and food security.
- AHRQ Methods Guide for Effectiveness of Comparative Effectiveness Reviews
- Centre for Reviews and Dissemination (Univ. of York) Includes guides on how to conduct systematic reviews and more.
- BEME: Best Evidence Medical Education
- Guidelines for Systematic Reviews of Health Promotion and Public Health This link will open a PDF document.
- << Previous: Review Types
- Next: What is a Rapid Review? >>
- Last Updated: Feb 24, 2023 11:02 AM
- URL: https://guides.temple.edu/systematicreviews
Temple University
University libraries.
See all library locations
- Library Directory
- Locations and Directions
- Frequently Called Numbers
Need help? Email us at [email protected]
IMAGES
VIDEO
COMMENTS
A good thesis statement is a single sentence contained in the introduction of a paper that provides the reader with some idea of what the writer is trying to convey in the body of the paper. The thesis statement is a condensed summary of th...
Statistical treatment in a thesis is a way of removing researcher bias by interpreting the data statistically rather than subjectively. Giving a thesis statistical treatment also ensures that all necessary data has been collected.
You’ve spent years preparing for your master’s degree or PhD. You’ve read, studied and spent hours of time and energy writing papers. Now you’ve arrived at the culmination of all this effort: writing your thesis.
Systematic reviewing requires time, skills, resources, and teamwork. Some systematic reviews may take much longer than running a primary study.
Systematic reviews (SRs) are a type of secondary research, which refers to the analysis of data that have already been collected through primary
A systematic review, on the other hand, is defined as a review using a systematic method to summarize evidence on questions with a detailed and
Systematic reviews typically answer their research question by synthesizing all available evidence and evaluating the quality of the evidence.
The distinction between a systematic literature review and a ... a systematic literature review in the context of a major research paper.
Steps for conducting a systematic literature review · Define a research question that is focused and narrow enough. · Identify keywords that will lead you to
A systematic review should provide an answer to the research question, it is not a broad overview of current trends and gaps in research. The review should
What is a Systematic Review? A literature review is often required as a precursor to (or part of) graduate work to inform further research activities and
A systematic review is a complex piece of research that aims to identify, select and synthesise all research published on a particular question or topic.
A systematic review question should be narrow in scope. The purpose of a systematic review is to draw conclusions based on the evidence to answer that one well-
that both synthesis research strategies of literature study and meta-study ... To guide the systematic review, the following main research