LARGE LANGUAGE MODELS IN SYSTEMATIC REVIEWS AND EVIDENCE SYNTHESIS
A TEMAC BIBLIOMETRIC MAPPING OF APPLICATIONS, PERFORMANCE AND AUTOMATION LEVELS
Keywords:
Large Language Models, systematic review, evidence synthesis, TEMAC, bibliometricsAbstract
The adoption of Large Language Models (LLMs) in systematic reviews and evidence synthesis is growing rapidly, but empirical evidence remains fragmented and concerns persist regarding reliability, reproducibility and performance. This study systematized and analysed this evidence, identifying applications, performance, benefits, limitations and automation levels. A mixed approach integrated the Consolidated Meta-Analytical Approach Theory (TEMAC) with the PRISMA protocol. Of 744 records retrieved from Scopus and Web of Science (2023–2026), 106 composed the bibliometric corpus and 95 full-text studies supported the qualitative synthesis. Applications concentrate on screening and data extraction; the GPT family predominates; and evaluation relies mainly on recall, precision, F1 and human agreement. Results indicate high recall, more variable precision and continued need for human supervision. The main contribution is an integrative model structured around tasks, technology, evaluation and governance.
References
AKINSELOYIN, Olalekan; JIANG, Xiaorui; PALADE, Vasile. A question-answering framework for automated abstract screening using large language models. Journal of the American Medical Informatics Association, [s. l.], v. 31, n. 9, p. 1939-1952, 2024. Disponível em: https://doi.org/10.1093/jamia/ocae166. Acesso em: 6 nov. 2026.
ALSHAMI, Ahmad; ELSAYED, Moustafa; ALI, Eslam; ELTOUKHY, Abddelrahman et al. Harnessing the power of ChatGPT for automating systematic review process: methodology, case study, limitations, and future directions. Systems, [s. l.], v. 11, n. 7, art. 351, 2023. Disponível em: https://doi.org/10.3390/systems11070351. Acesso em: 6 nov. 2026.
BANNACH-BROWN, Alexandra; PRIZYBYLA, Piotr; THOMAS, James; RICE, Andrew S. C. et al. Machine learning algorithms for systematic review: reducing workload in a preclinical review of animal studies and reducing human screening error. Systematic Reviews, [s. l.], v. 8, art. 23, 2019. Disponível em: https://doi.org/10.1186/s13643-019-0942-7. Acesso em: 6 nov. 2026.
BORAH, Rohit; BROWN, Andrew W.; CAPERS, Patrice L.; KAISER, Kathryn A. Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open, [s. l.], v. 7, n. 2, art. e012545, 2017. Disponível em: https://doi.org/10.1136/bmjopen-2016-012545. Acesso em: 6 nov. 2026.
BRAVERMAN, Harry. Labor and monopoly capital: the degradation of work in the twentieth century. New York: Monthly Review Press, 1974.
DAVENPORT, Thomas H.; KIRBY, Julia. Only humans need apply: winners and losers in the age of smart machines. New York: Harper Business, 2016.
GARTLEHNER, Gerald; KAHWATI, Leila; HILSCHER, Rainer; THOMAS, Ian et al. Data extraction for evidence synthesis using a large language model: a proof-of-concept study. Research Synthesis Methods, [s. l.], v. 15, n. 4, p. 576-589, 2024. Disponível em: https://doi.org/10.1002/jrsm.1710. Acesso em: 6 nov. 2026.
GUO, Eddie; GUPTA, Mehul; DENG, Jiawen; PARK, Ye-Jean et al. Automated paper screening for clinical reviews using large language models. Journal of Medical Internet Research, [s. l.], v. 26, art. e48996, 2024. Disponível em: https://doi.org/10.2196/48996. Acesso em: 6 nov. 2026.
HASAN, Bashar; SAADI, Samer; RAJJOUB, Noora S.; HEGAZI, Moustafa et al. Integrating large language models in systematic reviews: a framework and case study using ROBINS-I for risk of bias assessment. BMJ Evidence-Based Medicine, [s. l.], v. 29, n. 6, p. 394-398, 2024. Disponível em: https://doi.org/10.1136/bmjebm-2023-112597. Acesso em: 6 nov. 2026.
HIGGINS, Julian P. T.; THOMAS, James; CHANDLER, Jacqueline; CUMPSTON, Miranda et al. Cochrane handbook for systematic reviews of interventions. London: Cochrane, 2023. Disponível em: https://www.training.cochrane.org/handbook. Acesso em: 7 nov. 2026.
KHRAISHA, Qusai; PUT, Sophie; KAPPENBERG, Johanna; WARRAITCH, Azza et al. Can large language models replace humans in systematic reviews? Evaluating GPT-4’s efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages. Research Synthesis Methods, [s. l.], v. 15, n. 4, p. 616-626, 2024. Disponível em: https://doi.org/10.1002/jrsm.1715. Acesso em: 6 nov. 2026.
LANDIS, J. Richard; KOCH, Gary G. The measurement of observer agreement for categorical data. Biometrics, [s. l.], v. 33, n. 1, p. 159-174, 1977. Disponível em: https://doi.org/10.2307/2529310. Acesso em: 6 nov. 2026.
LI, Michael; SUN, Jianping; TAN, Xianping. Evaluating the effectiveness of large language models in abstract screening: a comparative analysis. Systematic Reviews, [s. l.], v. 13, art. 219, 2024. Disponível em: https://doi.org/10.1186/s13643-024-02609-x. Acesso em: 6 nov. 2026.
MAHMOUDI, Hesam; CHANG, Doris; LEE, Hannah; GHAFFARZADEGAN, Navid et al. Critical assessment of large language models’ (ChatGPT) performance in data extraction for systematic reviews: exploratory study. JMIR AI, [s. l.], v. 4, art. e68097, 2025. Disponível em: https://doi.org/10.2196/68097. Acesso em: 6 nov. 2026.
MARIANO, Ari Melo; ROCHA, Maíra Santos. Revisão da literatura: apresentação de uma abordagem integradora. In: AEDEM INTERNATIONAL CONFERENCE, 26., 2017, Reggio Calabria. Anais [...]. Reggio Calabria: AEDEM, 2017, p. 427-443. Disponível em: https://www.researchgate.net/publication/319547360_Revisao_da_Literatura_Apresentacao_de_uma_Abordagem_Integradora. Acesso em: 7 nov. 2026.
MARSHALL, Iain J.; WALLACE, Byron C. Toward systematic review automation: a practical guide to using machine learning tools in research synthesis. Systematic Reviews, [s. l.], v. 8, art. 163, 2019. Disponível em: https://doi.org/10.1186/s13643-019-1074-9. Acesso em: 6 nov. 2026.
MOTZFELDT JENSEN, Mette; BRIX DANIELSEN, Mathias; RIIS, Johannes; ASSIFUAH KRISTJANSEN, Karoline et al. ChatGPT-4o can serve as the second rater for data extraction in systematic reviews. PLoS ONE, [s. l.], v. 20, n. 1, art. e0313401, 2025. Disponível em: https://doi.org/10.1371/journal.pone.0313401. Acesso em: 6 nov. 2026.
NORDMANN, Kim; FISCHER, Florian. Capability of chatbots powered by large language models to support the screening process of scoping reviews: a feasibility study. BMC Health Services Research, [s. l.], v. 26, art. 109, 2026. Disponível em: https://doi.org/10.1186/s12913-025-13901-4. Acesso em: 6 nov. 2026.
OAMI, Takehiko; OKADA, Yohei; NAKADA, Taka-aki. GPT-3.5 Turbo and GPT-4 Turbo in title and abstract screening for systematic reviews. JMIR Medical Informatics, [s. l.], v. 13, art. e64682, 2025. Disponível em: https://doi.org/10.2196/64682. Acesso em: 6 nov. 2026.
OAMI, Takehiko; OKADA, Yohei; MAEDA, Kenjiro; NAKADA, Taka-aki. Performance of large language models and prompt engineering strategies for data extraction in systematic reviews. Frontiers in Digital Health, [s. l.], v. 8, art. 1799623, 2026. Disponível em: https://doi.org/10.3389/fdgth.2026.1799623. Acesso em: 6 nov. 2026.
O’MARA-EVES, Alison; THOMAS, James; McNAUGHT, John; MIWA, Makoto et al. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Systematic Reviews, [s. l.], v. 4, art. 5, 2015. Disponível em: https://doi.org/10.1186/2046-4053-4-5. Acesso em: 6 nov. 2026.
PAGE, Matthew J.; McKENZIE, Joanne E.; BOSSUYT, Patrick M.; BOUTRON, Isabelle et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ, [s. l.], v. 372, art. n71, 2021. Disponível em: https://doi.org/10.1136/bmj.n71. Acesso em: 6 nov. 2026.
REASON, Tim; BENBOW, Emma; LANGHAM, Julia; GIMBLETT, Andy et al. Artificial intelligence to automate network meta-analyses: four case studies to evaluate the potential application of large language models. PharmacoEconomics - Open, [s. l.], v. 8, n. 2, p. 205-220, 2024. Disponível em: https://doi.org/10.1007/s41669-024-00476-9. Acesso em: 6 nov. 2026.
SANGHERA, Rohan; HIRUNAVUKARASU, Arun James; EL KHOURY, Marc; O’LOGBON, Jessica et al. High-performance automated abstract screening with large language model ensembles. Journal of the American Medical Informatics Association, [s. l.], v. 32, n. 5, p. 893-904, 2025. Disponível em: https://doi.org/10.1093/jamia/ocaf050. Acesso em: 6 nov. 2026.
SOLLINI, Martina; PINI, Cristiano; LAZAR, Alexandra; GELARDI, Fabrizia; NINATTI, Gaia et al. Human researchers are superior to large language models in writing a medical systematic review in a comparative multitask assessment. Scientific Reports, [s. l.], v. 16, art. 173, 2025. Disponível em: https://doi.org/10.1038/s41598-025-28993-5. Acesso em: 6 nov. 2026.
SPILLIAS, Scott; OLLERHEAD, Katherine; ANDREOTTA, Matthew; ANNAND-JONES, Ruby et al. Evaluating generative AI for qualitative data extraction in community-based fisheries management literature. Environmental Evidence, [s. l.], v. 14, art. 9, 2025. Disponível em: https://doi.org/10.1186/s13750-025-00362-9. Acesso em: 6 nov. 2026.
SUSNJAK, Teo; HWANG, Peter; REYES, Napoleon H.; BARCZAK, Andre L. C. et al. Automating research synthesis with domain-specific large language model fine-tuning. ACM Transactions on Knowledge Discovery from Data, [s. l.], v. 19, n. 3, art. 68, 2025. Disponível em: https://doi.org/10.1145/3715964. Acesso em: 6 nov. 2026.
THIRUNAVUKARASU, Arun James; TING, Darren Shu Jeng; ELANGOVAN, Kabilan; GUTIERREZ, Laura et al. Large language models in medicine. Nature Medicine, [s. l.], v. 29, n. 8, p. 1930-1940, 2023. Disponível em: https://doi.org/10.1038/s41591-023-02448-8. Acesso em: 6 nov. 2026.
TRICCO, Andrea C.; LILLIE, Erin; ZARIN, Wasifa; O’BRIEN, Kelly K. et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Annals of Internal Medicine, [s. l.], v. 169, n. 7, p. 467-473, 2018. Disponível em: https://doi.org/10.7326/M18-0850. Acesso em: 6 nov. 2026.
VAN DIJK, Sanne H. B.; BRUSSE-KEIZER, Marjolein G. J.; BUCSÁN, Charlotte C.; VAN DER PALEN, Job et al. Artificial intelligence in systematic reviews: promising when appropriately used. BMJ Open, [s. l.], v. 13, n. 7, art. e072254, 2023. Disponível em: https://doi.org/10.1136/bmjopen-2023-072254. Acesso em: 6 nov. 2026.
VAN ECK, Nees Jan; WALTMAN, Ludo. Software survey: VOSviewer, a computer program for bibliometric mapping. Scientometrics, [s. l.], v. 84, n. 2, p. 523-538, 2010. Disponível em: https://doi.org/10.1007/s11192-009-0146-3. Acesso em: 6 nov. 2026.
YISHA, Zuhaer; OU, Peng; LI, Sheng; ZHANG, Lin et al. Large language models for data extraction in systematic reviews: a proof-of-concept study. BMC Medical Research Methodology, [s. l.], v. 26, art. 33, 2026. Disponível em: https://doi.org/10.1186/s12874-025-02729-5. Acesso em: 6 nov. 2026.
ZUBOFF, Shoshana. The age of surveillance capitalism: the fight for a human future at the new frontier of power. New York: PublicAffairs, 2019.