Abstract
-
Objective
This scoping review aimed to examine studies of nonagentic large language models (LLMs), LLM-based agents, and multiagent systems in emergency medicine and to identify current research trends and major gaps by analyzing their scope of clinical application, system structures, evaluation approaches, and input data characteristics.
-
Methods
The Web of Science, Scopus, PubMed, and CINAHL were searched for literature published from March 8, 2021, to March 7, 2026. English-language full-text studies published during this period were included if they addressed the application, evaluation, or benchmarking of nonagentic LLMs, LLM-based agents, or multiagent systems in emergency medicine or the emergency department. Additional studies were identified through reference tracking and supplementary searching. In total, 35 studies were analyzed.
-
Results
Of the 35 included studies, 26 were application studies, 6 were framework studies, and 3 were benchmark studies. Nonagentic LLMs were the most common system type (n=25), followed by LLM-based agents (n=7) and multiagent systems (n=3). Inputs were predominantly text-based, and evaluation mainly relied on expert comparison, retrospective record review, vignette-based comparison, and task-specific performance metrics. In contrast, workflow-level, prospective, and safety- and trustworthiness-oriented evaluations were limited.
-
Conclusion
LLMs in emergency medicine have shown potential for task-level decision support and documentation. However, the current literature remains focused on nonagentic LLM-based task support, whereas studies that reflect the dynamic workflow of real emergency departments remain limited. Future research should expand toward workflow-aware design, operational evaluation, multimodal data integration, multiagent-based role coordination, and validation of safety and trustworthiness.
-
Keywords: Emergency medicine; Emergency department; Large language models; Large language model–based agents; Multiagent systems
Capsule Summary
What is already known
In emergency medicine, the potential utility of large language models (LLMs) has been reported primarily for selected tasks such as triage, decision support, and documentation. However, reviews that jointly examine nonagentic LLMs, LLM-based agents, and mult-agent systems while accounting for the continuous workflow and time-dependent decision-making structure of the emergency department remain limited.
What is new in the current study
This study systematically mapped research on nonagentic LLMs, LLM-based agents, and multiagent systems in emergency medicine across the dimensions of clinical application, system characteristics, evaluation strategy, input data modality, and safety and trustworthiness. The findings indicate that the current literature is concentrated on a limited set of emergency department tasks and on nonagentic LLM-centered structures, while substantial gaps remain in workflow-level integration, agent-based design and orchestration, multimodal data use, time-sensitive data integration, and operational evaluation.
INTRODUCTION
The emergency department (ED) is a high-pressure clinical environment in which limited time, incomplete initial information, simultaneous patient arrivals, and rapidly changing patient conditions coexist. In this setting, clinicians must continuously perform tasks such as triage, diagnostic and treatment decision-making, handoff, discharge, and documentation, all of which can create substantial cognitive and operational burdens [
1–
4]. Accordingly, emergency medicine is a clinical domain in which rapid, context-adaptive decision support and workflow-level assistance are particularly important.
Recently, an increasing number of studies have explored the use of large language models (LLMs) to support selected tasks in emergency care [
5,
6]. Some studies have reported the potential utility of LLMs for tasks such as generating clinical recommendations, assessing severity, predicting admission, and drafting handoff and discharge documentation. Others have suggested their applicability to ED decision support, imaging recommendations, patient guidance, and diagnostic support. These early studies generally focused on individual tasks without iterative interaction with the environment; that is, they primarily involved the use of nonagentic LLMs.
More recently, LLM-based agents have been developed to support interactive information gathering and task execution. An LLM agent is a system that uses an LLM to autonomously plan, reason through complex tasks, use memory, and access external tools to achieve specific goals [
7–
10]. By contrast, multiagent systems involve two or more LLM-based agents with distinct roles, such as a triage agent, a diagnostic agent, and a documentation agent, that collaborate through coordinated interaction to accomplish complex tasks, thereby enabling role division and iterative communication that a single agent cannot achieve alone. These approaches may be better suited to ED settings characterized by pronounced temporal dynamics. Furthermore, some studies have extended this line of work to multiagent systems intended to reflect the division of roles and collaborative structure of the ED at the system level (
Fig. 1) [
11–
15]. Approaches involving collaboration among multiple agents with distinct roles, such as a triage agent and a charting agent, have also been reported [
12–
17].
The ED is an environment in which sequential decision-making, repeated information updates, interaction across roles, and workflow-level support are tightly interconnected, and decisions regarding testing, diagnosis, and disposition affect not only patient outcomes but also resource allocation and operational efficiency [
18]. Nevertheless, most current studies have focused on a limited set of tasks, such as triage, diagnostic or disposition support, and documentation using nonagentic LLMs. In addition, inputs have been largely text-centered, and evaluation has mainly relied on expert comparison, retrospective record review, vignette-based comparison, and task-specific performance metrics [
5,
6].
Several reviews have described recent trends and potential applications of LLM-based agents and multiagent systems [
19–
22]. However, these studies have mainly focused on general medical scenarios, and reviews that jointly examine nonagentic LLMs, LLM-based agents, and multiagent systems in the ED context remain limited.
Accordingly, this scoping review aimed to systematically organize studies on nonagentic LLMs, LLM-based agents, and multiagent systems in emergency medicine and to identify their application scope and key characteristics. To this end, the review included not only studies involving ED patients but also simulation and benchmark studies reflecting the ED environment. The core research question was as follows: In emergency medicine and ED settings, to which clinical tasks have these systems been applied, and what are their characteristics in terms of system type, evaluation strategy, input modality, and safety and trustworthiness? In addition, this review aimed to identify recurrent trends, major gaps, and implications for future LLM-based system design in emergency medicine.
METHODS
This scoping review was conducted and reported in accordance with the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) checklist (
Suppl. 1) [
23].
Studies were included if they met all of the following criteria: (1) they were directly related to emergency medicine or the ED setting; (2) they addressed the application, evaluation, or benchmarking of LLMs, LLM-based agents, or multiagent systems; (3) they involved ED-related tasks that were directly integrated into, or had clear implications for, ED clinical workflow; (4) they were published in English and available in full text; and (5) they were published from March 8, 2021, to March 7, 2026, a period during which LLM-based approaches emerged and became prominent.
Studies were excluded if they met one or more of the following criteria: (1) the context of emergency medicine or the ED was unclear, or ED data were used solely as a data source for disease-specific prediction or analysis without clear relevance to ED clinical workflow; (2) they addressed traditional machine learning, rule-based systems, or general clinical decision support without an LLM-based structure; (3) they were not original research, including review articles, editorials, commentaries, letters to the editor, conference abstracts, case reports, guidelines, or webpages; (4) they were not published in English or were not available in full text; or (5) they were published before March 8, 2021.
Information sources and search strategy
A comprehensive literature search was conducted using the Web of Science, Scopus, PubMed, and CINAHL. The search strategy was developed using key terms related to emergency medicine, LLMs, LLM-based agents, and multiagent systems and was adapted to the search syntax and indexing structure of each database (
Suppl. 2). The final search was performed on March 7, 2026.
To identify relevant studies that were not captured through database searching, the reference lists of candidate included studies were reviewed, and additional manual searches of related topics were conducted as needed.
To ensure objectivity in study selection, two researchers independently reviewed titles, abstracts, keywords, and, when necessary, full texts according to the eligibility criteria. The search and selection process was documented using a PRISMA flow diagram (
Fig. 2), and bibliographic information and screening records for retrieved studies were managed using Zotero (Digital Scholar), a reference management software. Disagreements arising during the review process were resolved through discussion between the two researchers, and when consensus could not be reached, a final decision was made after additional review.
Data charting for the included studies was performed using predefined extraction items. Two researchers reviewed the main characteristics of each study and organized information relevant to the study objectives in a structured format (
Table 1) [
24–
58]. Extracted items included study role, ED task, input data modality, primary output or function, system type, added features, and evaluation approach. Full texts were rereviewed as needed to supplement the extracted information.
In addition, because studies with the same clinical objective could differ in design and interpretation according to input data type, system structure, and evaluation approach, these items were categorized to facilitate cross-study comparison.
Synthesis and data analysis
To reduce heterogeneity in terminology across studies and ensure consistent classification, this review established operational definitions in advance that prioritized actual functional structure over author-reported labels. Structures that performed tasks without external tool calling or iterative execution were classified as nonagentic LLMs, including not only single-model applications but also noninteractive pipeline-type LLM applications. In contrast, structures that performed tasks while interacting with the environment through retrieval, tool use, planning, memory, or iterative action were classified as LLM-based agents. Structures in which two or more agents engaged in explicit communication and coordination based on role division were defined as multiagent systems. When boundaries were ambiguous, actual functional structure was prioritized over author-reported naming, and rule-based processing, static auxiliary modules, and simple retrieval augmentation were distinguished from agentic behavior and recorded separately as added features.
Study role was classified into three categories: application study, framework study, and benchmark study. Application studies were defined as studies whose primary contribution was the evaluation of performance or usefulness for a specific emergency medicine task. Framework studies were defined as studies whose primary contribution was the design and implementation of system structure, orchestration, and tool integration methods. Benchmark studies were defined as studies whose primary contribution was the development of datasets and benchmark or benchmark-style evaluation frameworks, or the systematic comparison of models. When boundaries were unclear, final classification was determined on the basis of the study’s primary objective and main contribution.
The studies were then narratively synthesized across the categories of clinical applications, system characteristics, evaluation and benchmarking, data modality, and safety and emerging directions. In this process, comparisons focused on relevance to ED tasks and workflows, the composition of system types and added features, the static and dynamic characteristics of evaluation, and feasibility for actual clinical application.
RESULTS
A total of 475 studies were initially identified through database searching. Of these, 274 underwent title- and abstract-level screening, and 41 were selected for full-text review. Thirty-two studies were included through the primary screening process, and three additional relevant studies were identified through reference-list review. Ultimately, 35 studies were included in the final analysis (
Fig. 2).
Among the 35 studies, application studies were the most common (n=26), followed by framework studies (n=6) and benchmark studies (n=3). In terms of system type, nonagentic LLMs were the most common (n=25), followed by LLM-based agents (n=7) and multiagent systems (n=3). Inputs were predominantly text-centered, and evaluation mainly consisted of expert comparison, retrospective real-world record evaluation, and vignette-based comparison. In contrast, more extensive evaluation designs, such as prospective evaluation, robustness assessment, and workflow-level evaluation, were identified in only a few studies. The characteristics and classification results of the individual studies are presented in
Table 1 [
24–
58], and the current evidence and limitations are summarized in
Table 2 [
24–
32,
34–
58].
Studies of LLMs in emergency medicine were concentrated on a limited set of tasks, including triage, diagnostic and treatment decision support, patient guidance, and documentation (
Table 1) [
24–
58]. Within the decision support category, recurrent tasks included ED acuity assessment, diagnostic or disposition support, and outpatient guidance [
24–
31,
36–
38,
48]. For example, several studies evaluated LLMs in a decision support role by presenting clinical vignettes or structured patient data, thereby simulating the function of a clinical decision support system. However, integration with electronic medical records (EMRs) was rarely reported; most studies used curated or synthetic inputs rather than live clinical data streams. In the documentation category, the major applications included handoff note drafting, discharge note generation, discharge summary drafting, and documentation automation [
46,
47,
55]. This distribution indicates that current LLM research in emergency medicine is focused primarily on supporting specific clinical tasks rather than addressing the entire ED process as a single integrated system.
The dominant pattern in current LLM research in emergency medicine was an output-centered structure based on nonagentic LLMs, whereas studies adopting agent-based or multiagent structures were relatively uncommon. Overall, many included studies were designed around nonagentic LLMs that generated outputs such as classifications, suggestions, predictions, or document drafts, indicating that recommendation-oriented design remained central at the system level (
Table 1) [
24–
58].
In contrast, studies incorporating added features or more complex components were relatively limited [
30,
32,
45,
46,
50–
58]. Among these studies, some took the form of LLM-based agents or multiagent systems through the incorporation of retrieval, retrieval-augmented generation (RAG; combining model outputs with external knowledge sources), handbook grounding, web search, RxNorm API integration (US National Library of Medicine), and agent coordination [
30,
32,
50–
58]. For example, Han and Choi [
54], Lu et al. [
52], and Sharma et al. [
50] presented multiagent structures involving role division and coordination, whereas Moser et al. [
55], Glicksberg et al. [
30], Karamanlıoğlu et al. [
51], and Niset et al. [
32] incorporated retrieval-based or RAG-based augmentation structures. Taken together, the current literature was centered on recommendation-oriented or generative structures based on nonagentic LLMs, whereas studies of agentic architectures incorporating role division, tool integration, and multistep processing remained relatively limited.
Evaluation strategies have become increasingly diverse; however, they remained centered overall on expert comparison, case-based comparison, retrospective record review, and task-specific performance evaluation. Triage-related studies were often evaluated comparatively against widely used validated triage systems or actual ED triage cases [
24,
25,
29,
31,
36,
37,
39–
43,
49–
53,
56–
58]. Studies related to diagnosis or disposition generally used retrospective designs centered on physician comparison, discharge-diagnosis comparison, retrospective case evaluation, or comparison with actual disposition [
26,
27,
30,
32,
34,
35,
38,
51,
57]. In documentation studies, quality was evaluated using methods such as ROUGE (recall-oriented understudy for gisting evaluation; a lexical overlap-based metric), BERTScore (bidirectional encoder representations from transformers score; a semantic similarity-based metric), 4C (coherence, consistency, correctness, and completeness) evaluation, and physician review [
45–
47,
55].
Meanwhile, studies classified as benchmark studies presented more structured evaluation frameworks, including dataset-level benchmarking, baseline comparison, distribution shift, and robustness to missing data [
56–
58]. This pattern indicates that evaluation in emergency medicine LLM research is being developed in increasingly diverse ways beyond simple accuracy comparisons.
Nevertheless, many studies remained at the stage of validating model performance within static cases or limited task settings. Although some designs were closer to real-world application, such as prospective multicenter evaluation [
33], evaluations that broadly reflected the overall ED workflow or operational context were still rare.
Input data were concentrated largely in text-based formats, whereas studies using both structured data and multimodal input were relatively limited. Many studies used text-based inputs such as triage summaries, electronic health record (EHR) notes, clinical vignettes, discharge-related text, and handoff notes [
24–
29,
32,
34–
41,
46–
49,
52–
55]. Overall, the current literature therefore reflects a heavily text-dominant input environment.
Some studies also used structured variables such as vital signs, laboratory results, and triage metadata [
30,
31,
42,
43,
45,
50,
51,
56–
58], whereas only a small number incorporated voice input [
33] or combined image and text input [
44]. To date, broad integration of speech, imaging, and structured data has been reported only to a limited extent.
Safety and trustworthiness were explicitly addressed in only some studies, whereas the overall evaluation was oriented more strongly toward task performance and output quality. Prior work has discussed safety, deferral, and trustworthy agent behavior [
59–
61], but only a limited number of studies directly evaluated these elements in the context of emergency medicine. For example, individual studies reported on the safety of lay-rescuer guidance for basic and advanced life support [
28], triage accuracy and reproducibility [
29], omissions or hallucinations in documentation output [
47], and patient safety–focused physician review [
45]. In contrast, most studies presented performance-centered metrics such as accuracy, agreement, recommendation quality, and documentation qual [
24–
30,
32–
49,
51–
56,
58].
Although some studies introduced agent-based or multiagent structures [
50,
52,
54], few explicitly evaluated safety-related behavior, uncertainty handling, traceability, and human oversight in the ED context. Thus, although the literature has expanded beyond nonagentic LLM-based task support to include retrieval-augmented, agent-based, and multiagent directions, studies directly validating safety and trustworthiness in the ED remain limited. A dimensional summary of these findings is presented in
Table 2 [
24–
32,
34–
58].
DISCUSSION
Summary of evidence
By organizing studies on nonagentic LLMs, LLM-based agents, and multiagent systems reported in the context of emergency medicine and the ED, this scoping review identified overall trends in the current literature with respect to application scope, system structure, evaluation strategy, input data modality, and safety and trustworthiness. The included literature consisted primarily of single task-specific application studies using nonagentic LLMs and focused on a limited set of tasks, such as triage, diagnostic and treatment decision support, patient guidance, and documentation. Input data were predominantly restricted to a single modality, typically text, and evaluations were often limited to narrow, task-specific performance metrics.
This pattern indicates that the current literature remains at a stage of exploring the applicability of LLMs to individual tasks that are relatively comparable and easy to evaluate, rather than addressing the ED as a complex system as a whole. Indeed, many studies examined relatively well-bounded tasks, which may represent a natural developmental pathway during the early stage of evidence accumulation. However, actual ED operations cannot be explained solely by the generation of a correct answer at a single time point; rather, the ED is an environment in which reassessment in response to changes in patient condition, information updates, external tool integration, and coordination across roles occur continuously [
2]. Therefore, although the current literature demonstrates the potential for prediction or generation at the level of individual tasks, it still provides only limited evidence regarding continuous workflow support and action-oriented system design.
At the system level, current research has been dominated by static recommendation or generation structures based on nonagentic LLMs. In contrast, studies of agents and multiagent systems involving retrieval, tool use, planning, iterative execution, and role-based coordination have been limited, and research on workflow-level orchestration tailored to the ED remains at an early stage. In other words, LLM research at the system level has continued to be characterized predominantly by output-centered design.
Evaluation mainly relied on case-based comparison, expert agreement, task-specific performance, and benchmark-based evaluation. Time sensitivity, resource constraints, escalation, and traceability, all of which are critical in real-world ED operations, were not adequately reflected. In addition, consideration of safety and trustworthiness was limited. Although some studies addressed the safety of lay-rescuer guidance, triage accuracy and reproducibility, documentation omissions or hallucinations, and patient safety–focused physician review, performance-centered metrics remained dominant overall, and only a small number of studies explicitly evaluated uncertainty handling, traceability, and human oversight in the ED context.
In terms of input data, text-based input was dominant. Many studies used relatively curated text inputs, such as triage summaries, EHR notes, clinical vignettes, handoff notes, and discharge-related text, whereas studies integrating structured data, multimodal input, and time-sensitive signals were relatively limited. Given the expanding use of multimodal generative artificial intelligence (AI) [
61], further investigation of multimodal and structured data integration in emergency medicine is warranted.
Taken together, current LLM research in emergency medicine shows meaningful potential at the level of individual tasks, but important gaps remain in continuous workflow support, dynamic system design, operational-level evaluation, multimodal data integration, and the assurance of safety and trustworthiness required for real ED application.
Implications for future ED-specific LLM system design
The gaps identified in this review have implications that extend beyond simply indicating the need for more studies and instead offer more specific guidance regarding the directions in which ED-specific LLM systems should be designed (
Fig. 3). First, in terms of clinical application, support structures are needed that extend beyond an individual task-centered approach to provide continuous support from patient reassessment through disposition planning, handoff, and discharge preparation. The current literature has shown a degree of potential for judgment and document generation at specific time points; however, because real-world ED workflows operate in the context of changing patient conditions and ongoing work-process continuity, future systems need to reflect workflow continuity more actively. Beyond these research gaps, practical barriers to real-world ED deployment also warrant attention. These include response latency in time-critical triage scenarios, regulatory considerations surrounding AI-assisted clinical decision-making, and technical challenges involved in integrating LLM outputs into existing EMR workflows.
From the perspective of system structure, a shift is warranted away from nonagentic LLM-centered design toward approaches that reflect workflow-aware agents, dynamic orchestration, and role coordination. For example, a role-based multiagent architecture in which a triage agent handles initial acuity assessment, a diagnostic agent integrates clinical reasoning, and a documentation agent generates structured notes could enable workflow-aware support while maintaining human oversight at key decision points [
15]. Pilot implementations in controlled ED settings, combined with RAG linked to EMR data, may represent a feasible starting point for validating such approaches [
55]. In the ED, multiple roles, including triage nurses, emergency physicians, consulting clinicians, and discharge coordinators, are involved, and decisions are iteratively revised on the basis of test results, patient responses, and interprofessional communication. Therefore, LLM-based agents and multiagent systems should be considered not merely technical extensions but potentially suitable structures for more closely reflecting actual clinical workflows and role division.
Changes are also needed in evaluation frameworks. The current literature mainly relies on task-specific accuracy and retrospective comparisons. However, to determine feasibility for ED implementation, operational evaluation, real-world validation, and safety-aware benchmarking need to be strengthened. In particular, a useful ED system is not simply one with a high correctness rate; it must also be possible to evaluate when the system should intervene, when it should defer to humans, and how incorrect recommendations affect the overall workflow.
From a data perspective, it is important to move beyond text-centered input toward temporal data integration, multimodal fusion, and the use of structured ED data. Real-world ED data are characterized by the combination of multiple modalities over time, including vital signs, physician orders, laboratory results, imaging, voice interactions, bedside conversations, and nursing observations. Therefore, future research needs to expand beyond a simple text-dominant paradigm toward approaches that jointly reflect temporality and multimodality.
Finally, safety and trustworthiness should be treated not as auxiliary considerations but as core design elements of emergency medicine-specific LLM systems. The ED is a high-risk environment, and even small errors can have cascading effects on patient safety, resource allocation, waiting times, and clinician workload. Therefore, designs and validation frameworks that explicitly incorporate traceability, uncertainty handling, and human oversight are needed. These elements will become even more important as agent-based structures continue to expand.
Limitations
This review has several limitations. First, because only English-language full-text studies were included, non-English literature, gray literature, technical reports, conference abstracts, and web-based materials were excluded, and some relevant studies may therefore have been missed.
Second, although the search included the Web of Science, Scopus, PubMed, and CINAHL, the diversity of terminology used in emergency medicine and LLM-based systems research, as well as differences in database indexing, may have limited search sensitivity. In particular, some studies may have incorporated agentic structures without explicitly describing them as “agent,” “LLM-based agent,” or “multi-agent.”
Third, to distinguish among nonagentic LLMs, LLM-based agents, and multiagent systems, this review applied operational definitions based on actual functional structure rather than author-reported labels. However, the recent literature uses concepts such as RAG, tool-calling pipelines, planning-based systems, memory-augmented workflows, role-based prompting, and multi-LLM collaboration in overlapping and inconsistent ways, so the same system may be classified differently depending on the level of interpretation.
Fourth, in the process of organizing heterogeneous elements such as ED task, primary input data, primary output or function, system type, added features, and evaluation approach into a comparable framework, some information was necessarily abstracted. As a result, the complex structure or detailed context of some studies may have been simplified.
Fifth, in accordance with the purpose of a scoping review, this review focused on mapping the research landscape and identifying conceptual gaps and did not perform a formal critical appraisal or meta-analysis of the included studies. Therefore, the findings should be interpreted not as evidence for quantitatively judging the effectiveness or superiority of individual systems, but rather as a synthesis of the current scope, structure, methodological trends, and limitations of the literature.
Finally, because many of the included studies were based on retrospective evaluation, simulated scenarios, curated benchmarks, or vignette-based assessment, evidence directly demonstrating prospective implementation, real-time operational performance, patient safety, and workflow impact in actual ED settings remains limited.
Conclusions
This scoping review showed that although LLM research in emergency medicine and ED settings is expanding rapidly, the current literature remains focused largely on task-level support based on nonagentic LLMs. In contrast, studies of LLM-based agents and multiagent systems that reflect the dynamic workflows and operational complexity of real EDs remain limited. Existing research demonstrates meaningful potential at the level of individual tasks, such as triage, diagnostic and treatment decision support, patient guidance, and documentation. However, key elements required for real-world ED implementation, including continuous support, dynamic orchestration, multimodal data integration, operational evaluation, and validation of safety and trustworthiness, have not yet been adequately addressed. Future research on emergency medicine-specific LLM systems needs to move beyond individual task-level performance toward integrated design and evaluation frameworks that reflect real clinical workflows, operational contexts, and safety and trustworthiness requirements.
NOTES
-
Author contributions
Conceptualization: all authors; Data curation: HK, SJ; Formal analysis: HK, SJ; Funding acquisition: MHL; Investigation: HK, SJ; Methodology: HK, SJ, MHL; Supervision: MHL, DHC; Visualization: HK, SJ; Writing–original draft: HK, SJ; Writing–review & editing: all authors. All authors read and approved the final manuscript.
-
Conflicts of interest
The authors have no conflicts of interest to declare.
-
Funding
This study was supported by a National Research Foundation of Korea (NRF) grant funded by the Korean Ministry of Science and ICT (No. RS-2026-25482499). The funders played no role in study design, data collection, analysis and interpretation of data, or the writing of this manuscript.
-
Data availability
The data analyzed in this study are available from the corresponding author upon reasonable request.
Supplementary materials
Suppl. 1.
PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) checklist.
ceem-26-136-Suppl-1.pdf
Fig. 1.Conceptual evolution of large language model (LLM)-based systems in emergency medicine. ED, emergency department.
Fig. 2.PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram.
Fig. 3.Proposed directions for future emergency department (ED) large language model (LLM) systems.
Table 1.Characteristics of included studies on LLMs, LLM-based agents, and multiagent systems in emergency medicine
Table 1.
|
Study |
ED task |
Input data (modality) |
Primary output/function |
System type |
Added feature |
Evaluation approach |
|
Application study |
|
|
|
|
|
|
|
Haim et al. [24] (2024) |
ESI-based triage acuity assessment |
Triage case data from EHR (text) |
Triage acuity decision |
Nonagentic LLM |
None |
Expert comparison with real ED patient cases |
|
Wang et al. [25] (2025) |
MEWS-based triage and outpatient guidance |
Simulated patient cases for triage and guidance (text) |
Triage scoring and outpatient department guidance |
Nonagentic LLM |
None |
Task-specific multimetric evaluation |
|
Williams et al. [26] (2024) |
Admission, radiology, and antibiotic decisions |
ED clinical notes (text) |
Admission, imaging, and antibiotic recommendations |
Nonagentic LLM |
None |
Note-based evaluation with physician comparison |
|
Del Monte et al. [27] (2025) |
Pediatric ED diagnosis support |
Pediatric ED clinical vignettes (text) |
Diagnostic support/candidate diagnosis generation |
Nonagentic LLM |
None |
LLM comparison with physician comparison and diagnostic accuracy scoring |
|
Bushuven et al. [28] (2023) |
Pediatric emergency diagnosis and first-aid guidance |
Pediatric BLS/PALS case vignettes (text) |
Lay-rescuer diagnostic and first-aid guidance |
Nonagentic LLM |
None |
Analysis of diagnostic accuracy and lay-rescuer guidance safety |
|
Franc et al. [29] (2024) |
CTAS triage acuity assignment |
Validated adult ED triage vignettes (text) |
CTAS triage level assignment |
Nonagentic LLM |
None |
Repeatability, reproducibility, and triage accuracy assessment |
|
Glicksberg et al. [30] (2024) |
Admission prediction |
ED triage notes and structured EHR data (text + structured) |
Admission prediction |
LLM-based agent |
RAG, few-shot examples |
Retrospective evaluation of zero-shot, few-shot, and RAG GPT-4 (OpenAI) |
|
Lafuente et al. [31] (2025) |
Automated emergency triage |
Hospital health data for automated triage (text + structured) |
Automated emergency triage classification |
Nonagentic LLM |
None |
Benchmark evaluation on MIMIC-IV-ED |
|
Niset et al. [32] (2025) |
Diagnostic prediction |
ED cases with early clinical data (text) |
Top 5 diagnostic prediction |
LLM-based agent |
Retrieval (textbook grounding) |
Real-world ED case evaluation |
|
Pasli et al. [33] (2025) |
Voice-assisted ED triage |
Voice entered triage data (text + voice) |
Triage area assignment support |
Nonagentic LLM |
None |
Prospective multicenter comparison with emergency medicine specialist reference |
|
Pasquadibisceglie et al. [34] (2025) |
Early admission prediction |
Early ED journey event logs as narrative text (text) |
Hospital admission prediction |
Nonagentic LLM |
None |
Real event log evaluation of admission and explainability analysis |
|
Wang et al. [35] (2025) |
Differential diagnosis generation |
Retrospective ED physician notes from initial presentation (text) |
Ranked differential diagnosis generation |
Nonagentic LLM |
None |
Preliminary comparison of ChatGPT (OpenAI) model iterations |
|
Williams et al. [36] (2024) |
Adult ED pairwise acuity assessment |
Adult ED presenting histories from physician notes (text) |
Higher-acuity patient identification |
Nonagentic LLM |
None |
Cross-sectional pairwise accuracy comparison against ESI-defined ground truth |
|
Arslan et al. [37] (2025) |
ESI-based triage |
Real-time ED clinical vignettes (text) |
ESI triage classification |
Nonagentic LLM |
None |
Prospective comparison with triage nurses and emergency physician reference |
|
Arslan et al. [38] (2025) |
Primary and differential diagnosis generation |
ED clinical vignettes (text) |
Primary and differential diagnosis generation |
Nonagentic LLM |
None |
Comparison with emergency physician and discharge diagnoses |
|
Fraser et al. [39] (2023) |
Diagnosis and triage recommendation |
ED patient symptom data (text) |
Diagnosis and triage recommendation |
Nonagentic LLM |
None |
Comparison with symptom checkers, physician review, and final ED diagnoses |
|
Lee et al. [40] (2025) |
KTAS 3–5 triage support |
ED triage note (text) |
Non-critical urgency classification |
Nonagentic LLM |
None |
Real-world conversation-based comparison against nurse |
|
Masanneck et al. [41] (2024) |
MTS-based triage level assignment |
Anonymized ED case vignettes (text) |
MTS triage level assignment |
Nonagentic LLM |
None |
Comparative evaluation against professional MTS raters |
|
Nedos et al. [42] (2026) |
ESI-based ED triage classification |
Anonymized ED case vignettes from 39,375 EMR cases (text + structured) |
ESI triage classification, clinic referral, and admission prediction |
Nonagentic LLM |
None |
Retrospective large-scale evaluation of multiple LLMs in 39,375 patients |
|
Zaboli et al. [43] (2025) |
MTS-based ED triage classification |
ED triage records with history and vital signs (text + structured) |
MTS triage code assignment |
Nonagentic LLM |
None |
Retrospective comparison with nurse triage using agreement, ROC, and clinical outcome metrics |
|
Günay et al. [44] (2024) |
ECG interpretation |
ECG case images with diagnostic prompts (image + text) |
ECG diagnosis/interpretation |
Nonagentic LLM |
None |
Comparative evaluation against cardiologists and emergency medicine specialists |
|
Hartman et al. [45] (2024) |
Handoff documentation |
ED notes, vitals, labs, and orders (text + structured) |
Handoff note drafting |
Nonagentic LLM |
Rule-based extraction |
ROUGE/BERTScore/SCALE plus physician patient safety–focused evaluation |
|
Song et al. [46] (2025) |
Discharge note generation |
ED initial record and discharge-related EHR inputs (text) |
Discharge note drafting |
Nonagentic LLM |
Rule-based postprocessing |
Blinded 4C evaluation with writing-time comparison |
|
Williams et al. [47] (2025) |
Discharge summary drafting |
Full ED clinician notes (text) |
Discharge summary drafting |
Nonagentic LLM |
None |
Expert physician review of discharge summaries for inaccuracies, hallucinations, and omissions |
|
Shekhar et al. [48] (2025) |
Ambulance request prioritization |
Ambulance request types (text) |
Ambulance request prioritization |
Nonagentic LLM |
None |
Agreement evaluation on ambulance request prioritization |
|
Thotapalli et al. [49] (2025) |
Youth mental health emergency triage |
Youth psychiatric emergency vignettes (text) |
Risk, admission, and urgency assessment |
Nonagentic LLM |
None |
Comparative vignette analysis with clinician agreement, expert review |
|
Framework study |
|
|
|
|
|
|
|
Sharma et al. [50] (2026) |
ESI-based emergency triage classification |
Symptom histories, vital signs, and clinical findings (text + structured) |
Collaborative ESI classification with clinical reasoning |
Multiagent system |
Agent coordination + workflow staging |
Preliminary framework evaluation using ESI handbook case studies |
|
Karamanlıoğlu et al. [51] (2025) |
Emergency triage and infectious disease triage support |
MIMIC-III-ED triage notes, vitals, and labs (text + structured) |
Privacy-preserving triage, diagnosis, and next-step support |
LLM-based agent |
Retrieval privacy modules |
USMLE benchmark plus expert reviewed MIMIC triage case study |
|
Lu et al. [52] (2024) |
ESI-based clinical triage classification |
Clinical triage case documents (text) |
Collaborative ESI-level triage decision support |
Multiagent system |
Agent coordination + ESI handbook retrieval |
Benchmark-based multiagent triage evaluation with human comparison |
|
Shen et al. [53] (2025) |
ESI-based emergency triage classification |
Emergency triage cases and clinical narratives (text) |
ESI-level triage classification |
Nonagentic LLM |
Continued pretraining + instruction tuning |
Performance comparison against prompt engineered, ML, and clinician baselines |
|
Han and Choi [54] (2025) |
KTAS triage and treatment planning |
Simulated ED scenarios from the Asclepius dataset (text) |
Role-based triage, treatment planning, and disposition support |
Multiagent system |
Agent coordination, web search, RxNorm API (NLM) |
Expert review of Asclepius scenarios with single agent baseline comparison |
|
Moser et al. [55] (2025) |
Emergency documentation automation |
Synthetic EMS dialogues and ED records (text) |
RAG-based information extraction for documentation |
LLM-based agent |
RAG pipeline |
RAG extraction performance evaluation |
|
Benchmark study |
|
|
|
|
|
|
|
Lee et al. [56] (2026) |
ED triage |
Triage records, handbook/counterfactual variants (text + structured) |
Triage acuity prediction |
LLM-based agent |
Retrieval (similar triage examples) |
Benchmark across datasets, distribution shift, missing data |
|
Gaber et al. [57] (2025) |
Triage, referral, and diagnosis support |
MIMIC-IV-ED symptoms, present illness, initial vitals (text + structured) |
Triage level, specialty recommendation, and diagnosis prediction |
LLM-based agent |
Retrieval (PubMed abstract RAG) |
Benchmark evaluation with clinician validation |
|
Solaiman et al. [58] (2026) |
Emergency triage deterioration prediction |
MIMIC-IV-ED records with vitals, notes, and labs (text + structured) |
ICU transfer or mortality risk prediction |
LLM-based agent |
Schema alignment/feature harmonization |
Benchmark evaluation across hospital and mass casualty incident-like regimes with machine learning |
Table 2.Summary of current evidence and research gaps in LLM research for emergency medicine
Table 2.
|
Dimension |
What current studies mainly show |
What remains limited or underexplored |
Reference |
|
Clinical application |
Research concentrates on a limited set of ED tasks, especially triage, diagnostic/disposition support, patient guidance, and documentation. |
Most studies focus on isolated tasks rather than integrated support across multiple ED stages. |
[24–31,36–38,45–48,55] |
|
System characteristic |
Most studies use single-LLM, output-oriented designs, while only a subset incorporates retrieval, RAG, coordination, or other augmented components. |
Agent-based and multiagent architectures remain a minority, and more complex workflow-aware orchestration is still limited. |
[24–31,36–40,43–49] |
|
Evaluation and benchmarking |
Evaluation is mainly based on expert comparison, retrospective review, case/vignette-based comparison, and task-specific metrics, with benchmark-style evaluation emerging in some studies. |
Prospective, multicenter, and broader workflow-level evaluation remain relatively uncommon. |
[24,25,27–31,34–39,41,42,44,46–48,53,55] |
|
Data modality |
Most studies rely on text-based ED inputs, sometimes supplemented with structured variables such as vital signs, laboratory data, or triage metadata. |
Voice- or image-inclusive inputs are rare, and broader multimodal integration remains limited. |
[24–25,27–32,35–36,38–40,44–45,47–50,53] |
|
Safety and trustworthiness |
Only a limited subset of studies explicitly addresses safety-related issues, while most emphasize task performance and output quality. Recent work also shows growing use of retrieval-augmented, agent-based, and multiagent structures. |
Explicit evaluation of safety-related behavior, uncertainty handling, traceability, human oversight, and ED-specific validation of agentic systems remains limited. |
[28–30,32,39,45,47–49,51–58] |
REFERENCES
- 1. Petrella RJ. The AI future of emergency medicine. Ann Emerg Med 2024;84:139-53.
- 2. Piliuk K, Tomforde S. Artificial intelligence in emergency medicine. A systematic literature review. Int J Med Inform 2023;180:105274.
- 3. Kang CY, Yoon JH. Current challenges in adopting machine learning to critical care and emergency medicine. Clin Exp Emerg Med 2023;10:132-7.
- 4. Okada Y, Ning Y, Ong ME. Explainable artificial intelligence in emergency medicine: an overview. Clin Exp Emerg Med 2023;10:354-62.
- 5. Preiksaitis C, Ashenburg N, Bunney G, et al. The role of large language models in transforming emergency medicine: scoping review. JMIR Med Inform 2024;12:e53787.
- 6. Hwai H, Ho YJ, Wang CH, Huang CH. Large language model application in emergency medicine and critical care. J Formos Med Assoc 2025;124:696-9.
- 7. Sapkota R, Roumeliotis KI, Karkee M. AI agents vs. agentic AI: a conceptual taxonomy, applications and challenges. Inf Fusion 2026;126:103599.
- 8. Cheng Y, Zhang C, Zhang Z, et al. Exploring large language model-based intelligent agents: definitions, methods, and prospects. Wiley Interdiscip Rev Data Min Knowl Discov 2026;16:e70111.
- 9. Xi Z, Chen W, Guo X, et al. The rise and potential of large language model based agents: a survey. Sci China Inf Sci 2025;68:121101.
- 10. Yao S, Zhao J, Yu D, et al. ReAct: synergizing reasoning and acting in language models. 11th International Conference on Learning Representations, ICLR 2023; 2023 May 1-5; Kigali, Rwanda.
- 11. Li X, Wang S, Zeng S, Wu Y, Yang Y. A survey on LLM-based multi-agent systems: workflow, infrastructure, and challenges. Vicinagearth 2024;1:9.
- 12. Ke Y, Yang R, Lie SA, et al. Mitigating cognitive biases in clinical decision-making through multi-agent conversations using large language models: simulation study. J Med Internet Res 2024;26:e59439.
- 13. Yao Z, Yu H. A survey on LLM-based multi-agent AI hospital [Preprint]. OSF; [posted 2025 Mar 2]. https://doi.org/10.31219/osf.io/bv5sg_v1
- 14. Kim Y, Park C, Jeong H, et al. A demonstration of adaptive collaboration of large language models for medical decision-making [Preprint]. arXiv:2411.00248; [posted 2024 Oct 31]. https://doi.org/10.48550/arXiv.2411.00248
- 15. Tang X, Zou A, Zhang Z, et al. MedAgents: large language models as collaborators for zero-shot medical reasoning. In: Ku LW, Martins A, Srikumar V, editors. Findings of the Association for Computational Linguistics: ACL 2024. Association for Computational Linguistics; 2024. p. 599–621.
- 16. Schmidgall S, Ziaei R, Harris C, Reis E, Jopling J, Moor M. AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments [Preprint]. arXiv:2405.07960; [posted 2024 May 13]. https://doi.org/10.48550/arXiv.2405.07960
- 17. Jiang Y, Black KC, Geng G, et al. MedAgentBench: a virtual EHR environment to benchmark medical LLM agents. NEJM AI 2025;2; (9).
- 18. Kim M, Lee S, Choi M, et al. Factors that predict emergency department length of stay in analysis of national data. Clin Exp Emerg Med 2025;12:35-46.
- 19. Liu F, Niu Y, Zhang Q, et al. A foundational architecture for AI agents in healthcare. Cell Rep Med 2025;6:102374.
- 20. Gorenshtein A, Omar M, Glicksberg BS, Nadkarni GN, Klang E. AI agents in clinical medicine: a systematic review [Preprint]. medRxiv 2025.08.22.25334232; [posted 2025 Aug 26]. https://doi.org/10.1101/2025.08.22.25334232
- 21. Wang W, Ma Z, Wang Z, et al. A survey of LLM-based agents in medicine: how far are we from Baymax?; In: Che W, Nabende J, Shutova E, Pilehvar MT, editors. Findings of the Association for Computational Linguistics: ACL 2025. Association for Computational Linguistics; 2025. p.10345-59.
- 22. Berikol GB, Kanbakan A, Ilhan B, Doğanay F. Mapping artificial intelligence models in emergency medicine: a scoping review on artificial intelligence performance in emergency care and education. Turk J Emerg Med 2025;25:67-91.
- 23. Tricco AC, Lillie E, Zarin W, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med 2018;169:467-73.
- 24. Haim GB, Saban M, Barash Y, et al. Evaluating large language model-assisted emergency triage: a comparison of acuity assessments by GPT-4 and medical experts. J Clin Nurs 2024;Nov; 28; [Epub]. https://doi.org/10.1111/jocn.17490
- 25. Wang C, Wang F, Li S, et al. Patient triage and guidance in emergency departments using large language models: multimetric study. J Med Internet Res 2025;27:e71613.
- 26. Williams CY, Miao BY, Kornblith AE, Butte AJ. Evaluating the use of large language models to provide clinical recommendations in the emergency department. Nat Commun 2024;15:8236.
- 27. Del Monte F, Barolo R, Circhetta M, et al. Diagnostic efficacy of large language models in the pediatric emergency department: a pilot study. Front Digit Health 2025;7:1624786.
- 28. Bushuven S, Bentele M, Bentele S, et al. "ChatGPT, can you help me save my child's life?": diagnostic accuracy and supportive capabilities to lay rescuers by ChatGPT in prehospital basic life support and paediatric advanced life support cases: an in-silico analysis. J Med Syst 2023;47:123.
- 29. Franc JM, Cheng L, Hart A, Hata R, Hertelendy A. Repeatability, reproducibility, and diagnostic accuracy of a commercial large language model (ChatGPT) to perform emergency department triage using the Canadian Triage and Acuity Scale. CJEM 2024;26:40-6.
- 30. Glicksberg BS, Timsina P, Patel D, et al. Evaluating the accuracy of a state-of-the-art large language model for prediction of admissions from the emergency room. J Am Med Inform Assoc 2024;31:1921-8.
- 31. Lafuente C, Rahim M. Evaluating large language models on hospital health data for automated emergency triage. Int J Comput Assist Radiol Surg 2025;20:1941-52.
- 32. Niset A, Melot I, Pireau M, et al. Grounded large language models for diagnostic prediction in real-world emergency department settings. JAMIA Open 2025;8:ooaf119.
- 33. Pasli S, Yadigaroğlu M, Kirimli EN, et al. ChatGPT-supported patient triage with voice commands in the emergency department: a prospective multicenter study. Am J Emerg Med 2025;94:63-70.
- 34. Pasquadibisceglie V, Appice A, Malerba D, Fiameni G. Leveraging a large language model (LLM) to predict hospital admissions of emergency department patients. Expert Syst Appl 2025;287:128224.
- 35. Wang J, Shue K, Liu L, Hu G. Preliminary evaluation of ChatGPT model iterations in emergency department diagnostics. Sci Rep 2025;15:10426.
- 36. Williams CY, Zack T, Miao BY, et al. Use of a large language model to assess clinical acuity of adults in the emergency department. JAMA Netw Open 2024;7:e248895.
- 37. Arslan B, Nuhoglu C, Satici MO, Altinbilek E. Evaluating LLM-based generative AI tools in emergency triage: a comparative study of ChatGPT Plus, Copilot Pro, and triage nurses. Am J Emerg Med 2025;89:174-81.
- 38. Arslan B, Erdem B, Osoydan Satıcı M, Nuhoğlu Ç, Soylu HY. Large language models in generating differential diagnoses in the emergency department: a comparative study of ChatGPT, Copilot, and emergency physician. Eurasian J Emerg Med 2025;24:201-7.
- 39. Fraser H, Crossland D, Bacher I, Ranney M, Madsen T, Hilliard R. Comparison of diagnostic and triage accuracy of Ada Health and WebMD symptom checkers, ChatGPT, and physicians for patients in an emergency department: clinical data analysis study. JMIR Mhealth Uhealth 2023;11:e49995.
- 40. Lee S, Jung S, Park JH, Cho H, Moon S, Ahn S. Performance of ChatGPT, Gemini and DeepSeek for non-critical triage support using real-world conversations in emergency department. BMC Emerg Med 2025;25:176.
- 41. Masanneck L, Schmidt L, Seifert A, et al. Triage performance across large language models, ChatGPT, and untrained doctors in emergency medicine: comparative study. J Med Internet Res 2024;26:e53297.
- 42. Nedos I, Zagalioti SC, Kofos C, et al. Is artificial intelligence ready for emergency department triage? A retrospective evaluation of multiple large language models in 39,375 patients at a university emergency department. J Clin Med 2026;15:1512.
- 43. Zaboli A, Brigo F, Brigiari G, et al. Chat-GPT in triage: still far from surpassing human expertise: an observational study. Am J Emerg Med 2025;92:165-71.
- 44. Günay S, Öztürk A, Yiğit Y. The accuracy of Gemini, GPT-4, and GPT-4o in ECG analysis: a comparison with cardiologists and emergency medicine specialists. Am J Emerg Med 2024;84:68-73.
- 45. Hartman V, Zhang X, Poddar R, et al. Developing and evaluating large language model-generated emergency medicine handoff notes. JAMA Netw Open 2024;7:e2448723.
- 46. Song JW, Park J, Kim JH, You SC. Large language model assistant for emergency department discharge documentation. JAMA Netw Open 2025;8:e2538427.
- 47. Williams CY, Bains J, Tang T, et al. Evaluating large language models for drafting emergency department encounter summaries. PLOS Digit Health 2025;4:e0000899.
- 48. Shekhar AC, Kimbrell J, Saharan A, Stebel J, Ashley E, Abbott EE. Use of a large language model (LLM) for ambulance dispatch and triage. Am J Emerg Med 2025;89:27-9.
- 49. Thotapalli S, Yilanli M, McKay I, et al. Potential of ChatGPT in youth mental health emergency triage: comparative analysis with clinicians. PCN Rep 2025;4:e70159.
- 50. Sharma K, Sivadas H, Edwards C, Gunja N, Reddy S. ED-triage-agent: a multi-agent framework for human-in-the-loop emergency triage. Int J Med Inform 2026;221:106652.
- 51. Karamanlıoğlu A, Demirel B, Tural O, Doğan OT, Alpaslan FN. Privacy-preserving clinical decision support for emergency triage using LLMs: system architecture and real-world evaluation. Appl Sci 2025;15:8412.
- 52. Lu M, Ho B, Ren D, Wang X. TriageAgent: towards better multi-agents collaborations for large language model-based clinical triage; In: Al-Onaizan Y, Bansal B, Chen YN, editors. Findings of the Association for Computational Linguistics: EMNLP 2024. Association for Computational Linguistics; 2024. p.5747-64.
- 53. Shen Q, Zhang X, Ren H, Guo Q, Yi Z. Knowledge-embedded large language models for emergency triage. Knowl Based Syst 2025;318:113431.
- 54. Han S, Choi W. Development of a large language model-based multi-agent clinical decision support system for Korean Triage and Acuity Scale (KTAS)-based triage and treatment planning in emergency departments. Adv Artif Intell Mach Learn 2025;5:3261-75.
- 55. Moser D, Bender M, Sariyar M. A pipeline for automating emergency medicine documentation using LLMs with retrieval-augmented text generation. Appl Artif Intell 2025;39:2519169.
- 56. Lee J, Shang T, Baik JY, et al. From promising capabilities to pervasive bias: assessing large language models for emergency department triage. J Healthc Inform Res 2026;10:689-709.
- 57. Gaber F, Shaik M, Allega F, et al. Evaluating large language model workflows in clinical decision support for triage and referral and diagnosis. NPJ Digit Med 2025;8:263.
- 58. Solaiman K, Sebastian J, Tobden K. Benchmarking early deterioration prediction across hospital-rich and MCI-like emergency triage under constrained sensing. 2026 IEEE 14th International Conference on Healthcare Informatics (ICHI); 2026 Jun 1-3; Minneapolis, MN, USA. IEEE; 2026; 1116-25.
- 59. Strong J, Men Q, Noble JA. Trustworthy and practical AI for healthcare: a guided deferral system with large language models. Proc AAAI Conf Artif Intell 2025;39:28413-21.
- 60. Hua W, Yang X, Jin M, et al. TrustAgent: towards safe and trustworthy LLM-based agents. In: Al-Onaizan Y, Bansal B, Chen YN, editors. Findings of the Association for Computational Linguistics: EMNLP 2024. Association for Computational Linguistics; 2024. p. 10000–16.
- 61. Buess L, Keicher M, Navab N, Maier A, Tayebi Arasteh S. From large language models to multimodal AI: a scoping review on the potential of generative AI in medicine. Biomed Eng Lett 2025;15:845-63.
