Large Language Models for Clinical Information Extraction from EHRs: A Systematic Review of Evaluation and Implementation Evidence

PreprintPreprints.org 2026 · 1 min read
Unique studies included
179
140 published studies and 39 preprints
Median F1 in published studies
0.887
Against a human reference standard · n=103
Studies with external validation
25/153
Among assessable studies

Systematic review of 179 studies finds strong reported extraction performance but limited external validation, prospective evaluation and production deployment.

This systematic review examined how generative large language models (LLMs) have been evaluated for extracting clinical information from electronic health records (EHRs), what evidence supports implementation, and which gaps separate reported performance from clinical use. The review included published studies and preprints and synthesized the evidence narratively without meta-analysis.

Study design and scope

The authors identified 1,282 records and included 179 unique studies: 140 published studies and 39 preprints. The synthesis used each study's best result, with a sensitivity analysis using the lowest reported configuration. Risk of bias was appraised with an adapted QUADAS-2 framework.

What the review found

In published studies, the median F1 score for LLM-based extraction against a human reference standard was 0.887 (IQR 0.800–0.948; n=103). Among 67 studies that tested several configurations, the median fell from 0.890 for the best configuration to 0.630 for the lowest one. The median F1 among preprints was 0.855 (n=30).

Most studies did not yet provide the evidence needed for clinical implementation. Risk of bias was high in 107 studies, unclear in 71 and low in one. External validation was reported in 25 of 153 assessable studies, prospective design in 2 of 142 and production deployment in none of 124 studies with available information.

Interpretation

Reported performance depended substantially on task definition and on which model configuration authors selected for reporting. The review therefore cautions against ranking models across studies without accounting for these differences. Clinician participation frequently supported annotation or research assessment rather than use in clinical workflows.

The authors conclude that evidence for generative LLMs in clinical information extraction remains largely retrospective and technical. Task-specific external validation and prospective evaluation embedded in clinical workflows are needed before implementation. This is a preprint and has not been peer reviewed.

Results at a glance

The headline numbers from the paper, in plain language.

How the model performed

Higher is better

Data: Against a human reference standard · n=103 · Among assessable studies — Preprints.org 2026

Scores reported on a 0–1 scale (AUC, kappa) are drawn ×100.

Show the numbers as a table
MetricValueWhat it means
Unique studies included179140 published studies and 39 preprints
Median F1 in published studies0.887Against a human reference standard · n=103
Studies with external validation25/153Among assessable studies

How to cite

Katherine Monsalve, Natalia Castaño-Villegas, José Zea, Laura Velásquez. Large Language Models for Clinical Information Extraction from Electronic Health Records: A Systematic Review of Evaluation and Implementation Evidence. Preprints.org. 2026. https://www.preprints.org/manuscript/202611.2137

@article{monsalve2026large,
  title     = {{Large Language Models for Clinical Information Extraction from Electronic Health Records: A Systematic Review of Evaluation and Implementation Evidence}},
  author    = {Katherine Monsalve and Natalia Castaño-Villegas and José Zea and Laura Velásquez},
  journal   = {Preprints.org},
  year      = {2026},
  url       = {https://www.preprints.org/manuscript/202611.2137},
  note      = {Preprint}
}

More research from Arkangel AI

All research

For hospitals and insurers

Validate these results at your institution

In a 30-minute conversation, we review whether this study fits your data, team and governance process.

Start a research partnershipGet new research by email
Read paperCite