This systematic review examined how generative large language models (LLMs) have been evaluated for extracting clinical information from electronic health records (EHRs), what evidence supports implementation, and which gaps separate reported performance from clinical use. The review included published studies and preprints and synthesized the evidence narratively without meta-analysis.
Study design and scope
The authors identified 1,282 records and included 179 unique studies: 140 published studies and 39 preprints. The synthesis used each study's best result, with a sensitivity analysis using the lowest reported configuration. Risk of bias was appraised with an adapted QUADAS-2 framework.
What the review found
In published studies, the median F1 score for LLM-based extraction against a human reference standard was 0.887 (IQR 0.800–0.948; n=103). Among 67 studies that tested several configurations, the median fell from 0.890 for the best configuration to 0.630 for the lowest one. The median F1 among preprints was 0.855 (n=30).
Most studies did not yet provide the evidence needed for clinical implementation. Risk of bias was high in 107 studies, unclear in 71 and low in one. External validation was reported in 25 of 153 assessable studies, prospective design in 2 of 142 and production deployment in none of 124 studies with available information.
Interpretation
Reported performance depended substantially on task definition and on which model configuration authors selected for reporting. The review therefore cautions against ranking models across studies without accounting for these differences. Clinician participation frequently supported annotation or research assessment rather than use in clinical workflows.
The authors conclude that evidence for generative LLMs in clinical information extraction remains largely retrospective and technical. Task-specific external validation and prospective evaluation embedded in clinical workflows are needed before implementation. This is a preprint and has not been peer reviewed.




