Según el artículo Las revisiones sistemáticas (I) (y II), una revisión sistemática es:
Las revisiones sistemáticas son el mejor esfuerzo por recopilar y sintetizar evidencia científica sobre un tema, a través de un método que asegure que los sesgos y limitaciones a los que hemos hecho mención en el artículo precedente de esta serie, sean los mínimos posibles. Esto se logra utilizando los principios del método científico.
Las etapas de una revisión sistemática son:
- formulación de la pregunta de la revisión,
- definición de las fuentes y del método de búsqueda,
- definición y aplicación de criterios de selección de los estudios,
- análisis crítico de los estudios,
- extracción y síntesis de los resultados de los estudios,
- y conclusiones e inferencias.
En Systematic Review and Meta-analysis: When One Study Is Just not Enough se define una revisión sistemática como:
A systematic review uses a process to identify comprehensively all studies for a specific focused question (drawn from research and other sources), appraise the methods of the studies, summarize the results, present key findings, identify reasons for different results across studies, and cite limitations of current knowledge (8,9). In a systematic review, all decisions used to compile information are meant to be explicit, allowing the reader to gauge for him- or herself the quality of the review process and the potential for bias. In this way, systematic reviews tend to be more transparent than their narrative cousins, although they too can be biased if the selection or emphasis of certain primary studies is influenced by the preconceived notions of the authors or funding sources (10).
En What is a systematic review? se define una revisión sistemática como:
A systematic review summarises the results of available carefully designed healthcare studies (controlled trials) and provides a high level of evidence on the effectiveness of healthcare interventions. Judgments may be made about the evidence and inform recommendations for healthcare.
These reviews are complicated and depend largely on what clinical trials are available, how they were carried out (the quality of the trials) and the health outcomes that were measured. Review authors pool numerical data about effects of the treatment through a process called meta-analyses. Then authors assess the evidence for any benefits or harms from those treatments. In this way, systematic reviews are able to summarise the existing clinical research on a topic.
Una revisión sistemática se puede realizar a partir de un mapeo sistemático. Las diferencias que existen entre ambos tipos de estudios han sido analizadas en The value of mapping studies: a participant-observer case study (2010); en este artículo se presenta la siguiente tabla de diferencias entre un mapeo sistemático y una revisión sistemática:
| SLR process | Mapping Study | SLR |
|---|---|---|
| Research Question | General – related to research trends. Which researchers, how much activity, what type of studies etc. | Specific - related to outcomes of empirical studies. Of the form: Is technology/method A better or not than B? |
| Search process | Defined by topic area | Defined by research question |
| Search strategy requirements | Less stringent if only research trends are of interest | Extremely stringent – all relevant studies must be found |
| Quality evaluation | Not essential | Important to ensure that results are based on best quality evidence |
| Results | Set of papers related to a topic area and counts of the number of papers in various categories | Answer to specific research question, possible with qualifiers (e.g. results apply to novices only). |
En el artículo Systematic mapping studies in software engineering (2008), se explican algunas de las ventajas de un systematic review frente a un literature review:
Briefly, a systematic review (SR) goes through existing primary reports, reviews them in-depth and describes their methodology and results. Compared to literature reviews common in any research project, a SR has several benefits: a well-defined methodology reduces bias, a wider range of situations and contexts can allow more general conclusions, and use of statistical meta-analysis can detect more than individual studies in isolation (Kitchenham & Charters 2007).
La mayor desventaja es que un systematic review requiere un mayor esfuerzo:
However, SRs also have several drawbacks, the main one being that they require considerable effort.
En el artículo Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement, también disponible en Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement, se presenta PRISMA (preferred reporting items for systematic reviews and meta-analyses), un método para realizar revisiones sistemáticas. En este artículo, una revisión sistemática se define como:
A systematic review is a review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyze data from the studies that are included in the review. Statistical methods (meta-analysis) may or may not be used to analyze and summarize the results of the included studies.
Por otro lado, un metaanálisis se define como:
Meta-analysis refers to the use of statistical techniques in a systematic review to integrate the results of included studies.
El método PRISMA se compone de 27 elementos y un proceso de aplicación compuesto de 4 fases.
Después del artículo original de PRISMA, se han publicado otros más que desarrollan o extienden el método:
- The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration
- PRISMA for Abstracts: Reporting Systematic Reviews in Journal and Conference Abstracts
En Systematic Review and Meta-analysis: When One Study Is Just not Enough se define metaanálisis como:
Depending on the nature of the data, the results of a systematic review can be summarized in text or graphic form. In graphic form, it is common for different trials to be depicted in a plot where the point estimate and 95% confidence interval for each study are presented on an individual line (11). When results are mathematically combined (a process sometimes referred to as pooling), this is referred to as meta-analysis. Graphically, the pooled result is often presented as a diamond at the bottom of the plot.
When performing a meta-analysis, a review team usually combines aggregate-level data reported in each primary study (point and variance estimate of the summary measure).
Una revisión sistemática y un metaanálisis se pueden combinar en un mismo estudio, por ejemplo Web-Based Digital Health Interventions for Weight Loss and Lifestyle Habit Changes in Overweight and Obese Adults: Systematic Review and Meta-Analysis, cuyo método fue:
We searched PubMed or Medline, SciELO, Lilacs, PsychNet, and Web of Science up to July 2018, as well as references of previous reviews for randomized trials that compared Web-based digital health interventions to offline interventions. Anthropometric changes such as weight, body mass index (BMI), waist, and body fat and lifestyle habit changes in adults with overweight and obesity were the outcomes of interest. Random effects meta-analysis and meta-regression were performed for mean differences (MDs) in weight. We rated the risk of bias for each study and the quality of evidence across studies using the Grades of Recommendation, Assessment, Development, and Evaluation approach.
También existen los frameworks SALSA y ReSiste-CSH.
En A Guide to Conducting a Standalone Systematic Literature Review se define una guía compuesta de 8 pasos para realizar una revisión sistemática:
The guide I present here has eight major steps (see Figure 1) that one needs to take to conduct a systematic literature review. Indeed, these steps are very valuable for any kind of literature review; however, for a review to be scientifically rigorous, all of the following steps are essential.
- Identify the purpose: the first step in any review requires the reviewers to clearly identify the review’s purpose and intended goals, which is necessary for the review to be explicit to its readers.
- Draft protocol and train the team: for any review that employs more than one reviewer, reviewers need to be completely clear and in agreement about the procedure they will follow, which requires both a written, detailed protocol document and training for all reviewers to ensure consistency in how they execute the review.
- Apply practical screen: also called screening for inclusion, this step requires that the reviewers be explicit about what studies they considered for review and which ones they eliminated without further examination (a very necessary part of any literature review). For excluded studies, the reviewers must state their practical reasons for not considering them and justify how the resulting review can still be comprehensive given the practical exclusion criteria.
- Search for literature: the reviewers need to be explicit in describing the details of the literature search and need to explain and justify how they assured the search’s comprehensiveness.
- Extract data: after reviewers have identified all the studies that should be included in the review, they need to systematically extract the applicable information from each study.
- Appraise quality: also called screening for exclusion, the reviewers need to explicitly spell out the criteria they use to judge which papers they will exclude for insufficient quality. Researchers need to score all included papers, depending on the research methodologies they employ, for their quality.
- Synthesize studies: also known as analysis, this step involves combining the facts extracted from the studies by using appropriate techniques, whether quantitative, qualitative, or both.
- Write the review: in addition to the standard principles to be followed in writing research papers, the process of a systematic literature review needs to be reported in sufficient detail such that other researchers can independently reproduce the review’s results.
Revisión sistemática del estado de la cuestión (systematic literature review)
Revisiones bibliográficas sistematizadas: guía de recursos
How to... write a literature review
How to conduct a systematic or evidence-based literature review
Una revisión sistemática del estado de la cuestión es una revisión de la bibliografía referente a un tema realizada mediante un método que es explícito y transparente, que cumple un protocolo o un conjunto de fases.
Procedures for Performing Systematic Reviews
Cita: Barbara Kitchenham. Keele University and National ICT Australia Ltd, 2004.
Systematic review in software engineering
Cita: J. Biolchini,P.G. Mian, A.C.C. Natali, G.H. Travassos. System Engineering and Computer Science Department COPPE/UFRJ, Technical Report ES, 679(05), 45, May 2005.
Systematic Reviews in the Social Sciences: A Practical Guide
Cita: Mark Petticrew, Helen Roberts. Wiley-Blackwell, 2005.
Guidelines for performing Systematic Literature Reviews in Software Engineering
Cita: Barbara Kitchenham, Stuart Charters. Keele University and Durham University Joint Report, 2007.
Experiences Using Systematic Review Guidelines
Cita: Mark Staplesa, Mahmood Niazi. Journal of Systems and Software. Volume 80, Issue 9, September 2007, Pages 1425-1437.
Systematic literature reviews in software engineering: Preliminary results from interviews with researchers
Cita: M.A. Babar, He Zhang. 3rd International Symposium on Empirical Software Engineering and Measurement, Pages 346-355, 15-16 Oct. 2009.
A Guide to Conducting a Systematic Literature Review of Information Systems Research
Cita: Chitu Okoli, Kira Schabram. Sprouts: Working Papers on Information Systems, 10(26), 2010.
Systematic review in software engineering: where we are and where we should be going
Cita: Barbara A. Kitchenham. In Proceedings of the 2nd international workshop on Evidential assessment of software technologies (EAST '12). ACM, New York, NY, USA, 2012.
A Guide to Conducting a Standalone Systematic Literature Review
Cita: Okoli, C. Communications of the Association for Information Systems, 37, 2015.
Systematic Approaches to a Successful Literature Review
Cita: Andrew Booth, Anthea Sutton, Diana Papaioannou. Second Edition, Sage Publications, 2016.
Doing a Systematic Review
Cita: Angela Boland, Gemma Cherry, Rumona Dickson. Sage Publications, 2017.
An Introduction to Systematic Reviews
Cita: David Gough, Sandy Oliver, James Thomas. Second Edition, Sage Publications, 2017.
Ejemplos de revisión sistemática (en español)
Calidad en Cursos Abiertos Masivos y en Línea. Revisión de literatura del 2012 - 2016
Cita: Alejandra Meléndez, Mariela Román, Mar Pérez-Sanagustín, Jorge J. Maldonado. Actas de la Jornada de MOOCs en Español en EMOOCs 2017 (EMOOCs-ES 2017), Pages 72-80.
Los MOOC y la evaluación del alumnado: revisión sistemática (2012-2016)
Cita: María del Mar Sánchez Vera, Víctor González Calatayud, María Paz Prendes Espinosa. revista d'innovació educativa, 18, 2017.
Ejemplos de revisión sistemática (en inglés)
Requirements modeling languages for software product lines: A systematic literature review
Cita: Samuel Sepúlveda, Ania Cravero, Cristina Cachero. Information and Software Technology, Volume 69, January 2016. DOI: 10.1016/j.infsof.2015.08.007
A Systematic Analysis and Synthesis of the Empirical MOOC Literature Published in 2013–2015
Cita: George Veletsianos and Peter Shepherdson. International Review of Research in Open and Distributed Learning, Volume 17, Issue 2, February 2016.
A systematic review of research on the flipped learning method in engineering education
Cita: Aliye Karabulut-Ilgu, Nadia Jaramillo Cherrez, Charles T. Jahren. British Journal of Educational Technology, Volume 49, Issue 3, 2018. DOI: 10.1111/bjet.12548
Research challenges in accessible MOOCs: a systematic literature review 2008–2016
Cita: Sanchez-Gordon, S. & Luján-Mora, S. Universal Access in the Information Society, 17, 775-789, 2018. DOI: https://doi.org/10.1007/s10209-017-0531-2
Ver en línea: Research challenges in accessible MOOCs: a systematic literature review 2008–2016
Modeling and Management Big Data in Databases—A Systematic Literature Review
Cita: Diana Martinez-Mosquera, Rosa Navarrete, Sergio Lujan-Mora. Sustainability 2020, 12(2). https://doi.org/10.3390/su12020634
Rediscovering the use of chatbots in education: A systematic literature review
Cita: José Quiroga Pérez, Thanasis Daradoumis, Joan Manuel Marquès Puig. Computer Applications in Engineering Education. https://doi.org/10.1002/cae.22326
Empirical Studies on Web Accessibility of Educational Websites: A Systematic Literature Review
Cita: Milton Campoverde-Molina, Sergio Luján-Mora, Llorenç Valverde. IEEE Access, 8(1), p. 91676-91700, 2020. e-ISSN: 2169-3536. https://doi.org/10.1109/ACCESS.2020.2994288
Web-Based Serious Games and Accessibility: A Systematic Literature Review
Cita: Luis Salvador-Ullauri, Patricia Acosta-Vargas, Sergio Luján-Mora. Applied Sciences, 10(21), p. 1-25, 2020. ISSN: 2076-3417. https://doi.org/10.3390/app10217859
Systematic literature review on software architecture of educational websites
Cita: Milton Campoverde-Molina, Sergio Luján-Mora, Llorenç Valverde. IET Software, p. 1-21, 2021. ISSN: 1751-8814. https://doi.org/10.1049/sfw2.12024
Ejemplos de revisión (en inglés)
Educational Data Mining: A Review of the State of the Art
Cita: Cristóbal Romero, Sebastián Ventura. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews). Volume 40, Issue 6, November 2010, Pages 601-618.
Este artículo tiene 304 citas que ocupan casi 8 páginas de las 18 páginas que tiene.
Ejemplos de metaanalisis (en inglés)
Estos artículos son una revisión sistemática de revisiones sistemáticas. Se podría decir que son una "metarevisión sistemática". También se puede decir que son "tertiary research", frente a primary research y secondary research.
Systematic literature reviews in software engineering - A systematic literature review
Cita: Barbara Kitchenham, O. Pearl Brereton, David Budgen, Mark Turner, John Bailey, Stephen Linkman. Information and Software Technology. Volume 51, Issue 1, January 2009, Pages 7-15. DOI: 10.1016/j.infsof.2008.09.009
Systematic literature reviews in global software development: A tertiary study
Cita: J. M. Verner, O. P. Brereton, B. A. Kitchenham, M. Turner, M. Niazi. 16th International Conference on Evaluation & Assessment in Software Engineering (EASE 2012), Ciudad Real, 2012, Pages 2-11. DOI: 10.1049/ic.2012.0001
Relation between Alice software and programming learning: a systematic review of the literature and meta-analysis
Cita: Joana M. Costa, Guilhermina L. Miranda. (2017). British Journal of Educational Technology, 48(6), 1464-1474. DOI: 10.1111/bjet.12496
Does the medical literature remain inadequately described despite having reporting guidelines for 21 years? – A systematic review of reviews: an update
Cita: Jin, Y., Sanger, N., Shams, I., Luo, C., Shahid, H., Li, G., Bhatt, M., Zielinski, L., Bantoto, B., Wang, M., Abbade, L. P., Nwosu, I., Leenus, A., Mbuagbaw, L., Maaz, M., Chang, Y., Sun, G., Levine, M. A., Adachi, J. D., Thabane, L., … Samaan, Z. (2018). Journal of multidisciplinary healthcare, 11, 495-510. DOI: 10.2147/JMDH.S155103
Landscaping systematic mapping studies in software engineering: A tertiary study
Cita: Muhammad Uzair Khan, Salman Sherin, Muhammad Zohaib Iqbal, Rubab Zahid. (2019). Journal of Systems and Software, Volume 149, March 2019, Pages 396-436. DOI: 10.1016/j.jss.2018.12.018
Herramientas
- StArt (State of the Art through Systematic Review)
- SciMAT (Science Mapping Analysis Tool)
- NVivo
- Rayyan QCRI
- Publish or Perish
- Parsifal