Large Language Models in Healthcare: Navigating Promise and Limitations

Introduction: The LLM Revolution in Healthcare

Large language models (LLMs) are a class of artificial intelligence systems trained on vast corpora of text to predict and generate language. In practical terms, they can draft narratives, summarize complex information, extract key facts, and interact conversationally—capabilities that have accelerated their adoption across industries. In healthcare, the appeal is immediate: clinical work is information-dense, documentation-heavy, and operationally complex, creating fertile ground for healthcare AI to reduce friction and improve throughput.

Models such as GPT and related architectures have rapidly moved from experimental demonstrations to pilots in medical applications, including ambient clinical documentation, chart summarization, patient messaging support, and coding assistance. Yet healthcare is not “just another” domain for generative AI. The stakes are higher, the data are more sensitive, and regulatory expectations are more stringent. Healthcare leaders—clinical, operational, compliance, and IT—must therefore understand both the opportunities and the risks of LLMs before scaling implementation.

The central challenge is not whether large language models can generate useful output; they often can. The question is whether those outputs can be reliably and safely integrated into clinical workflows without introducing unacceptable risk. This requires balancing innovation with patient safety, medical ethics, and regulatory compliance. Effective deployment depends on governance, validation, and clear accountability—along with a realistic understanding of what LLMs do well (language) and what they do not inherently guarantee (truth, clinical reasoning, and responsibility).

This article outlines where LLMs are already creating value in healthcare AI, the limitations that must be addressed, and a practical approach to responsible adoption—grounded in current best practices and the evolving regulatory landscape.

The Promise: How LLMs Are Transforming Medical Applications

LLMs are primarily language engines, but language is a core substrate of healthcare: histories, assessments, plans, discharge instructions, prior authorizations, and claims all depend on text. When deployed thoughtfully, large language models can improve efficiency, consistency, and access—especially when paired with structured clinical data, workflow integration, and appropriate human oversight.

1) Clinical documentation and administrative burden reduction

Clinician burnout is strongly associated with administrative load, particularly documentation and EHR-related tasks. LLM-enabled tools can:

Draft encounter notes from clinician prompts or ambient transcripts (with safeguards)
Generate concise chart summaries for continuity of care
Suggest problem lists, medications, and follow-up plans based on available context
Support medical coding by proposing ICD-10-CM, CPT, and HCC candidates, then highlighting evidence in the chart
Draft prior authorization narratives and appeal letters aligned to payer requirements

These functions are attractive because they often fall into “language transformation” tasks: converting clinical conversations and chart content into structured documentation. When implemented properly, this can return time to clinicians and improve note completeness. For organizations deploying chart review and coding support, LLMs can also help standardize evidence extraction, reducing variability and helping clinical documentation integrity (CDI) teams focus on higher-value work.

In this operational space, companies such as Arkangel AI position LLM-assisted chart review and coding support as part of a broader healthcare AI strategy—where the value comes not only from generation, but from traceability, auditability, and workflow fit.

2) Diagnostic support and clinical decision-making assistance

LLMs are increasingly explored for decision support—summarizing differential diagnoses, suggesting next steps, and synthesizing patient history with guideline-based recommendations. Examples include:

Drafting differential diagnoses from a symptom narrative
Summarizing relevant guideline recommendations for a clinical scenario
Assisting with medication reconciliation narratives and contraindication reminders (when coupled to reliable drug databases)
Translating patient-provided histories into structured clinical problem representations

Used responsibly, LLMs can function as “clinical copilots” that reduce cognitive load and help clinicians consider overlooked possibilities. However, these use cases require more stringent safeguards because they influence clinical decisions. High-performing outcomes depend on access to accurate, up-to-date knowledge sources and the ability to cite evidence—ideally via retrieval-augmented generation (RAG) tied to trusted clinical references rather than free-form generation.

3) Patient engagement: chatbots, symptom checkers, and personalized communication

Patient engagement is another area where LLMs can deliver immediate value—especially for organizations dealing with staffing constraints, high message volumes, and health literacy gaps. Common applications include:

Automated drafting of portal message replies for clinician review
Post-discharge instructions tailored to a patient’s regimen and comprehension level
Appointment preparation and after-visit summaries
Basic symptom triage with clear guardrails and escalation rules
Chronic disease education (e.g., diabetes, asthma, heart failure) with culturally and linguistically appropriate messaging

When properly designed, these tools can improve responsiveness and patient experience while maintaining safety through conservative triage, disclaimer language, and escalation pathways. The best deployments treat LLMs as communication assistants, not autonomous clinicians.

4) Medical research acceleration: literature review, data synthesis, and insight generation

Medical knowledge doubles quickly, and clinicians and researchers face information overload. LLMs can help accelerate:

Literature screening and summarization for systematic reviews (with human verification)
Extraction of key findings and limitations across sets of papers
Drafting research protocols, study outlines, and statistical analysis plans (with domain expert oversight)
Hypothesis generation and early-stage drug discovery insights when paired with cheminformatics and structured biomedical datasets
Synthesis of real-world evidence narratives from de-identified datasets

These workflows benefit from LLM strengths—summarization, clustering concepts, and drafting coherent narratives—while still requiring rigorous methodological controls. In research contexts, transparency about model prompts, versions, and validation is essential for reproducibility.

5) Accessibility improvements: multilingual support and health literacy enhancement

LLMs can expand access by translating medical content into multiple languages and tailoring educational materials to different reading levels. Potential benefits include:

Multilingual discharge instructions and medication guidance
Plain-language explanations of diagnoses and procedures
Culturally adapted health education materials
Improved accessibility for patients with limited health literacy

These capabilities can support equity goals, but organizations must validate translations and educational outputs for accuracy, cultural appropriateness, and local standards of care. “Fluent” language is not the same as “clinically correct” language.

The Limitations: Critical Challenges Healthcare Leaders Must Address

The same characteristics that make LLMs compelling—flexible text generation and conversational interfaces—also introduce non-obvious risks. Leaders should view these limitations not as reasons to avoid LLMs entirely, but as design constraints requiring governance, validation, and workflow controls.

1) Hallucinations and accuracy concerns

A well-documented limitation of LLMs is hallucination: the generation of plausible-sounding but incorrect statements. In a clinical context, hallucinations can become safety events if they influence care decisions, documentation, or patient instructions.

Examples of healthcare-relevant hallucination risks include:

Fabricating citations, guidelines, or contraindications
“Filling in” missing clinical facts that were never documented
Misstating dosages, durations, or monitoring requirements
Overstating diagnostic certainty based on incomplete information

Even when an LLM is often correct, the tail risk matters. Healthcare organizations need to assume that errors will occur and design systems so those errors are detectable, containable, and unlikely to harm patients—particularly in clinical decision support and patient-facing use cases.

Mitigation approaches include RAG with trusted sources, constrained generation, confidence signaling (with careful interpretation), mandatory human review, and automated checks against structured data (e.g., medication lists, allergies, labs).

2) Training data limitations: bias, outdated information, and specialized gaps

LLMs learn patterns from training data. If the underlying data reflect historical inequities, incomplete representation, or outdated standards, outputs can perpetuate those shortcomings. Key risks include:

Bias in clinical recommendations or communication tone across demographic groups
Underperformance for rare diseases, pediatric populations, pregnancy, or complex comorbidities
Outdated clinical guidance (e.g., older screening intervals or deprecated therapies)
Limited context on local formularies, payer rules, or institutional protocols

Healthcare leaders should treat LLM outputs as hypotheses to be verified, not authoritative clinical truth. Specialized medical LLMs trained on curated clinical corpora may reduce some issues, but bias and drift remain concerns—particularly as clinical standards evolve.

3) Privacy and HIPAA compliance risks

Many LLM workflows involve processing protected health information (PHI): encounter narratives, labs, diagnoses, and identifiers. Privacy and security risks include:

Inappropriate data sharing with third-party model providers
Insufficient controls over data retention and model training on customer data
Prompt leakage (sensitive data exposed through logs, analytics, or vendor tooling)
Cross-tenant data exposure in multi-tenant environments if isolation is flawed
Inadequate access controls, audit trails, and monitoring

To maintain HIPAA compliance, organizations need strong contractual and technical safeguards, including Business Associate Agreements (BAAs) where applicable, encryption, role-based access, data minimization, and clear policies on retention. De-identification can help for some use cases, but many clinical workflows require identifiable data, making robust security architecture non-negotiable.

4) Lack of clinical reasoning: pattern matching vs. understanding

LLMs can emulate reasoning in text, but they do not “understand” medicine in the way clinicians do. They generate outputs based on learned statistical relationships between tokens, which can create the illusion of deep comprehension.

This limitation shows up when:

A case requires nuanced causal reasoning (e.g., distinguishing correlation vs. causation)
The correct answer depends on missing information that should trigger “ask a question” behavior
The model provides overconfident recommendations despite uncertainty
Safety depends on risk stratification, temporal reasoning, or precise guideline adherence

In high-stakes clinical decisions, LLMs should be treated as assistants that can summarize and propose, not arbitrate. Where possible, pairing LLMs with rule-based checks, validated calculators, or guideline engines can provide additional safety structure.

5) Liability and accountability: who is responsible when AI errs?

Healthcare organizations must determine accountability pathways before deploying LLMs at scale. Liability questions can arise when:

A clinician follows incorrect AI-generated advice
A patient is harmed after receiving AI-generated instructions
Documentation errors lead to billing inaccuracies or compliance findings
AI outputs contribute to delayed diagnosis or inappropriate treatment

Regulators and courts will look for reasonable safeguards: clear labeling, staff training, audit trails, human oversight, and evidence that the organization validated the system for intended use. Leaders should collaborate with legal, compliance, and risk management teams to define:

The intended use and prohibited use cases
The level of required human review
Documentation policies for AI-assisted content
Incident response and monitoring procedures
Vendor responsibilities and indemnification where appropriate

Practical Takeaways: Implementing LLMs Responsibly in Your Organization

Responsible implementation is less about “turning on” a model and more about designing a socio-technical system: governance, workflows, training, and continuous monitoring. The following actions are practical starting points for healthcare leaders evaluating LLM and GPT-style deployments.

Define intended use with clear boundaries (and write them down).
Specify whether the LLM is used for documentation drafting, chart summarization, coding support, patient messaging drafts, or clinical decision support. Explicitly prohibit high-risk autonomous behaviors (e.g., unsupervised prescribing recommendations).
Establish governance before deployment.
Create an AI governance structure that includes clinical leadership, compliance, privacy/security, IT, quality/safety, and operational stakeholders. Define approval gates, model change control, and periodic review cadence.
Start with low-risk administrative and documentation use cases.
Prioritize workflows where errors are unlikely to cause direct patient harm and where human review is standard (e.g., note drafting, coding suggestions, chart summarization for staff). Expand to clinical decision support only after demonstrating reliability and safety.
Build “human-in-the-loop” oversight into the workflow—not as an afterthought.
Require clinician or coder attestation for AI-generated content that enters the medical record or claim. Design user interfaces that make it easy to verify source evidence, not just accept fluent text.
Validate on local data and local workflows.
Test performance on representative patient populations, specialty mixes, and documentation styles. Evaluate not only average performance but also failure modes (rare diseases, pediatrics, limited data scenarios). Include equity-focused testing where feasible.
Use retrieval-augmented generation (RAG) and constrained outputs for clinical content.
Where clinical recommendations or guideline summaries are generated, ground the model in curated, current references (institutional protocols, drug databases, payer rules). Prefer outputs that cite sources and limit free-form generation.
Implement privacy-by-design and HIPAA-aligned controls.
Apply data minimization, strong access controls, encryption, audit logs, retention limits, and secure vendor contracts (including BAAs as appropriate). Ensure PHI handling is mapped end-to-end—from prompt capture to output storage.
Train staff on appropriate use, limitations, and escalation.
Education should cover hallucinations, bias, and when to override AI suggestions. Provide role-specific guidance (clinicians, nurses, coders, CDI, front desk) and embed training into onboarding and annual refreshers.
Monitor performance continuously and plan for incident response.
Establish KPIs (turnaround time, documentation quality, coding accuracy), safety metrics (near misses, harmful suggestions), and user feedback loops. Create a clear process for reporting issues, retraining, prompt updates, or rolling back features.
Select vendors based on transparency and healthcare readiness—not demos.
Evaluate explainability, auditability, security posture, model versioning, and evidence of validation. Ask how outputs are grounded, how PHI is handled, and how the vendor supports monitoring and change management. Solutions like Arkangel AI are typically evaluated on these operational criteria—how well they integrate into chart review and compliance workflows—not only on generative fluency.

Future Outlook: Where Healthcare AI Is Headed

The next phase of healthcare AI is likely to be defined by specialization, integration, and regulation—moving away from general-purpose chat and toward embedded, validated capabilities that are tightly aligned to clinical workflows.

Specialized medical LLMs trained on curated datasets

General LLMs are impressive, but healthcare requires domain specificity and reliability. The field is moving toward:

Medical LLMs trained on curated clinical text and biomedical literature
Fine-tuning for specialties (radiology, oncology, cardiology, emergency medicine)
Institutional adaptation using local templates, policies, and documentation norms
Improved grounding to reduce hallucination and improve traceability

Even with specialization, organizations should expect ongoing evaluation needs. Clinical knowledge changes, and models can drift as workflows evolve.

Deeper integration with EHRs and clinical workflows

Standalone chat interfaces are unlikely to be the long-term end state. Value increases when LLMs are integrated into:

EHR note workflows (drafting, summarizing, highlighting missing elements)
Inbasket and patient messaging (drafts with escalation pathways)
Coding and CDI tooling (evidence-linked code suggestions)
Care management and utilization management (summaries, criteria mapping)
Quality reporting and measure abstraction

However, integration raises the bar for governance: model outputs must be attributable, auditable, and consistent with documentation policies. Workflow design will increasingly determine success more than model selection.

Regulatory evolution and emerging standards

Regulation of AI in healthcare is evolving. In the U.S., FDA oversight of software as a medical device (SaMD) and clinical decision support continues to develop, alongside broader policy initiatives for trustworthy AI. Healthcare organizations should anticipate:

Increased expectations for transparency, intended use statements, and performance monitoring
Requirements for managing model updates and change control
Greater scrutiny of real-world performance and bias
More formalized best practices for validation and safety management

Leaders should track FDA guidance and relevant standards efforts, and align internal governance to these emerging expectations—even when a particular use case does not meet the threshold for FDA-regulated SaMD.

Multimodal AI combining text, imaging, waveforms, and genomics

Healthcare is inherently multimodal: radiology images, pathology slides, ECG waveforms, lab trends, and genomics data all contribute to decision-making. The future likely includes systems that combine:

Text (notes, guidelines, patient messages)
Imaging (radiology, dermatology, pathology)
Structured EHR data (labs, vitals, meds)
Omics data (genomics, proteomics) where available

Multimodal models could enable richer clinical summaries and more context-aware decision support. At the same time, they introduce new validation challenges: it becomes harder to understand why a model produced an output, and errors can propagate across modalities. This will intensify the need for transparency, auditability, and human oversight.

Conclusion: Charting a Thoughtful Path Forward

LLMs and GPT-like systems are reshaping healthcare AI by making language tasks—documentation, summarization, patient communication, and evidence extraction—faster and more scalable. In the best implementations, large language models reduce administrative burden, improve consistency, and support clinicians in navigating information overload. They also offer meaningful opportunities to improve accessibility through multilingual communication and health literacy optimization.

At the same time, the limitations are material: hallucinations, bias and outdated knowledge, privacy and HIPAA risks, and the absence of true clinical reasoning. These are not edge cases; they are predictable behaviors of probabilistic language models operating in a high-stakes environment. Healthcare leaders should therefore treat LLMs as powerful tools—but not replacements for clinical judgment, established clinical decision support, or rigorous quality systems.

Organizations that lead in this space will be those that balance innovation with caution: defining intended use, establishing governance, validating on local populations, maintaining human oversight, and partnering with vendors that prioritize healthcare-grade compliance and transparency. The next step is organizational readiness—assessing where LLMs can safely add value now, what guardrails are required, and how to build the operational muscle for continuous monitoring as the technology evolves.