Abstract
-
Objective
This study integrated a machine learning–based Score for Emergency Risk Prediction (SERP), developed using objective mortality endpoints, with the Patient Acuity Category Scale (PACS) and evaluated its predictive performance in clinical use.
-
Methods
This single-center, retrospective cohort study included all emergency department (ED) patients from a large tertiary hospital between January 1, 2018, and December 31, 2019. Using a reclassification framework, SERP was incorporated into PACS to derive two enhanced triage models. PACS+ model 1 down-triaged patients with low predicted 30-day mortality risk and up-triaged those with high risk. PACS+ model 2 up-triaged only high-risk patients, whereas low-risk patients retained their original category. Predictive performance in the test cohort was assessed using the area under the receiver operating characteristic curve (AUC) and decision curve analysis.
-
Results
The derivation cohort included 97,188 ED visits, and the test cohort included 97,212 ED visits. In the derivation set, the mean age of the patients was 58.97±18.41 years, and 47,993 (49.4%) were female. Overall, 19.9%, 57.4%, 22.5%, and 0.2% of patients were triaged to PACS categories P1–P4, respectively. The 30-day mortality rate was 2.8% in the derivation set and 2.7% in the test cohort. For 30-day mortality prediction, PACS+ model 1 (AUC, 0.828; 95% confidence interval [CI], 0.820–0.836) and PACS+ model 2 (AUC, 0.812; 95% CI, 0.805–0.818) outperformed PACS (AUC, 0.722; 95% CI, 0.714–0.729). PACS+ model 1 consistently achieved greater net benefit across the range of clinical thresholds.
-
Conclusion
Integrating machine learning–based SERP with PACS improved 30-day mortality prediction in ED triage.
-
Keywords: Emergency department triage; Machine learning; Emergency department; Artificial intelligence
Capsule Summary
What is already known
Emergency department overcrowding is a major global issue that requires accurate triage for appropriate resource allocation. Conventional triage systems rely heavily on triage providers’ clinical judgment and experience, which can result in variability, cognitive bias, and mis-triage. Machine learning (ML)-based triage models have shown superior predictive performance for outcomes such as mortality and intensive care unit admission. However, widespread clinical implementation of ML-based triage models remains limited because of model opacity, workflow integration barriers, and lack of clinician trust.
What is new in the current study
This study integrates an interpretable ML-based model, the Score for Emergency Risk Prediction (SERP), with a four-level clinical triage system used in Singapore, the Patient Acuity Category Scale (PACS), to create an enhanced, transparent triage tool, PACS+. PACS+ aims to improve the prediction of 30-day mortality while maintaining interpretability to support clinician trust and usability. This study is among the first to integrate interpretable ML models into emergency department triage, thereby helping bridge the gap between algorithmic development and real-world clinical application.
INTRODUCTION
Emergency department (ED) overcrowding is a significant global challenge [
1], making effective patient triage essential. Triage systems stratify patients by risk on arrival, enabling rapid identification of those requiring urgent intervention and optimization of resource allocation [
2]. Widely used scales such as the Emergency Severity Index, the Canadian Triage and Acuity Scale, and Singapore’s Patient Acuity Category Scale (PACS) provide a framework for this process [
3–
5]. However, these systems rely heavily on the clinical judgment of triage providers, leading to substantial interrater variability [
3,
6] and potential cognitive biases that may compromise patient outcomes through under- or over-triage [
7].
To address these limitations, machine learning (ML) models have been developed to provide more stable and accurate risk predictions for outcomes such as mortality, hospital admission, and intensive care unit (ICU) admission [
8–
11]. Research has consistently shown that ML models using information available at triage can outperform conventional five-level triage systems [
12,
13]. Despite their predictive power, implementation of these models in clinical practice remains limited [
6]. A key barrier is the “black box” nature of many ML algorithms [
14], which obscures their decision-making process. This lack of transparency can hinder trust and adoption among healthcare providers, who are essential to successful integration [
15].
Interpretable ML offers a promising solution by making model predictions understandable [
16]. A notable example is the Score for Emergency Risk Prediction (SERP), which uses only six routinely available triage variables to estimate a patient’s mortality risk [
8]. SERP has been externally validated and has demonstrated superior performance in predicting 2-, 7-, and 30-day mortality compared with other clinical scores [
17]. Building on this work, our study aimed to enhance the existing PACS triage system by integrating the interpretable SERP score. We created an enhanced framework, named PACS+, and evaluated whether it improved the prediction of 30-day mortality (primary outcome), as well as 2- and 7-day mortality (secondary outcomes), using a large real-world clinical dataset.
METHODS
Ethics statement
This study was approved by the Singapore Health Services Centralized Institutional Review Board (No. 2021/2122). A waiver of consent was granted for data collection and analysis because of the use of the deidentified data and the retrospective nature of the study.
Study design, setting, and population
This was a single-center, retrospective cohort study of patients seen in the Singapore General Hospital (SGH) ED. Singapore is a city-state in Southeast Asia, and SGH is the largest and oldest public tertiary hospital in the country. The SGH ED receives more than 120,000 visits and has 36,000 inpatient admissions annually. Electronic health record data were obtained from Singapore Health Services and analyzed [
18]. The study population included all adult patients seen from January 1, 2018, to December 31, 2019. The study period was intentionally restricted to the pre–COVID-19 period to minimize potential confounding effects introduced by the COVID-19 pandemic. Visits were excluded if patients were younger than 21 years, were noncitizens with incomplete medical records, or had missing data (mainly triage priority levels). Visits in 2018 were assigned to the derivation cohort, whereas visits in 2019 were assigned to the test cohort. Using the derivation cohort, we applied a reclassification concept and manually integrated SERP and PACS to derive PACS+. Each ED visit was treated as an independent encounter.
In this study, the primary outcome was 30-day mortality. We recognize that there is no universal objective standard for evaluating triage accuracy. We selected 30-day mortality as the primary outcome because existing literature has established it as a significant outcome for ED care [
19–
23], thereby facilitating comparability across studies. To capture short-term mortality, we also included 2- and 7-day mortality as secondary outcomes. These were not selected as primary outcomes because the expected number of events was very low. These outcomes were defined as all-cause mortality within 2, 7, and 30 days of the initial presentation to the ED, regardless of the place of death.
Patient Acuity Category Scale
The PACS is a symptom-based, four-level triage scale used by all hospitals in Singapore. PACS classifies patients into four priority levels. P1 patients are critically ill and require immediate management and resuscitation; P2 patients are generally nonambulatory, have severe symptoms, and require early medical attention; P3 patients are generally ambulatory and present with mild to moderate symptoms; and P4 patients represent non-emergency cases that can be managed in primary care (
Suppl. 1). PACS is assessed by trained triage nurses and provides a framework for assigning an appropriate triage level based on patients’ presenting complaints, vital signs, and the user’s clinical judgment. Depending on the assigned priority level, patients are allocated to different treatment areas in the ED for consultation with attending physicians. Resources are allocated according to patient acuity, such that 70% of physicians are assigned to P1 and P2 patients [
24].
Score for Emergency Risk Prediction
In the study by Xie et al. [
8], three SERP scores were developed based on 2-, 7-, and 30-day mortality among patients who presented to the ED. All three scores were derived using AutoScore [
25], an ML framework that generates an interpretable point-based clinical score and has been externally validated [
17]. Among the three, 30-day SERP achieved the most satisfactory performance for short- and long-term mortality compared with 2- and 7-day SERPs. It uses six variables available at triage—age, heart rate, respiratory rate, systolic and diastolic blood pressure, and cancer history [
8]—and was derived using 30-day mortality as the primary outcome (
Suppl. 2). The points for each component are summed to calculate the 30-day SERP score for each patient.
Patient characteristics
Baseline characteristics of the study population were analyzed in both the development and test cohorts. Categorical variables were reported as numbers (percentages), whereas continuous variables were presented as mean±standard deviation.
Model development and validation
For this analysis, we categorized patients into two groups based on their 30-day SERP score. Patients with 30-day SERP scores ≥26 were defined as high risk, whereas those with scores <26 were defined as low risk. A cutoff score of 26 was selected for 30-day SERP as the point nearest the upper-left corner of the receiver operating characteristic (ROC) curve, which maximizes the balance between sensitivity and specificity for predicting 30-day mortality [
8]. Within the development cohort, we applied the reclassification concept and manually integrated SERP and PACS, generating multiple candidate iterations of the enhanced triage system PACS+. We then selected two final versions: PACS+ model 1 (
Table 1) and PACS+ model 2 (
Table 2). PACS+ model 1 involves down-triaging patients with 30-day SERP scores <26 (low mortality risk) and up-triaging patients with 30-day SERP scores ≥26 (high mortality risk). For example, a patient originally triaged as P1 using PACS with a 30-day SERP score <26 would be down-triaged to P2 using PACS+ model 1. PACS+ model 2 takes a more cautious approach: patients with 30-day SERP scores ≥26 are up-triaged, whereas patients with 30-day SERP scores <26 remain in the same triage category. For example, a patient originally triaged as P1 using PACS with a 30-day SERP score <26 would not be down-triaged. When values needed to calculate SERP scores were missing, we imputed them using the median value, consistent with the approach used in the original SERP development study [
8].
After refinement of PACS with 30-day SERP, predictive performance was evaluated using the area under the ROC curve (AUC) in the test cohort. Bootstrapped samples were used to calculate 95% confidence intervals (CIs). AUCs were compared using the DeLong test for correlated ROC curves, and the differences with 95% CIs were calculated. The PACS+ models were designed as risk stratification tools that assign patients to discrete triage priority categories rather than produce continuous predicted probabilities. Therefore, conventional calibration metrics assessing concordance between observed and predicted probabilities were not applicable in this study. Instead, we described the distribution of observed outcome proportions across PACS+ triage categories. We also performed decision curve analysis [
26] for each model to evaluate clinical utility against two default strategies: “assuming all patients are triaged to P1 and P2 (treat all)” and “assuming no patients are triaged to P1 and P2 (treat none).” The y-axis represents the net benefit across threshold probabilities for different models. The x-axis reflects the threshold probability, indicating the triage provider’s tolerance for triaging potentially low-risk patients to higher P1 or P2. Net benefit incorporates information about the benefit of correctly triaging patients (true positives) and the cost of over-triage (false positives). The “treat all” curve initially shows a net benefit but declines and eventually crosses zero and becomes negative when the threshold approaches the actual incidence of the outcome. This occurs because the cost of treating false positives outweighs the benefit of true positives. The “treat none” curve consistently yields a net benefit of zero, serving as the baseline.
The threshold probability reflects the triage provider’s risk tolerance. For example, setting a threshold probability at 10% indicates that a clinician is willing to triage 9 of 10 potentially low-risk patients to higher P1 and P2 to avoid missing one true case. In contrast, a threshold probability of 80% indicates a preference to reduce false positives, whereby only patients with an 80% or higher predicted risk are triaged to higher priority levels, thereby reducing unnecessary interventions.
The decision curve analysis enables identification of the range of threshold probabilities for which a model is clinically useful, the magnitude of that benefit, and the optimal model among multiple alternatives. A bar chart was used to compare the primary and secondary outcomes achieved using PACS, PACS+ model 1, and PACS+ model 2. The data were analyzed using R ver. 4.3.1 (R Foundation for Statistical Computing).
RESULTS
Baseline characteristics of the study cohort
Between January 1, 2018, and December 31, 2019, a total of 264,854 ED visits were assessed. Of these, 70,454 visits were excluded because of age younger than 21 years (n=12,120), noncitizen status (n=46,168), or missing information (n=12,166), leaving 194,400 ED visits for analysis: 97,188 in the derivation cohort and 97,212 in the test cohort (
Fig. 1). Of the included visits, 119,626 were made by unique patients. In the derivation set, the mean age of patients presenting to the ED was 58.97±18.41 years, and 47,993 (49.4%) were female. The racial composition was similar to that of the general population (70.5% Chinese, 13.1% Malay, 12.1% Indian, and 4.3% others). Of the 97,188 ED visits, 19,302 (19.9%) were triaged as PACS category P1, 55,833 (57.4%) as P2, 21,846 (22.5%) as P3, and 207 (0.2%) as P4. A substantial majority of patients were triaged to PACS category P2, indicating that they were assessed as being in severe distress and requiring timely care. Overall, 16.1% of patients had 30-day SERP scores of ≥26 (high risk). The mortality rates observed in the derivation set were 497 (0.5%) at 2 days, 1,020 (1.1%) at 7 days, and 2,707 (2.8%) at 30 days, and the ICU admission rate was 0.9% (n=902).
Table 3 shows the baseline characteristics of the derivation and test sets, indicating similarity in age, sex, racial composition, comorbidities, and other characteristics.
Suppl. 3 presents additional baseline characteristics and demonstrates overall comparability between the two cohorts. Compared with the derivation set, patients in the test set had marginally lower 2-, 7-, and 30-day mortality and ICU admission rates, possibly reflecting improvements in healthcare over time.
The performance of PACS, PACS+ model 1, and PACS+ model 2, as assessed by ROC analysis in the test cohort, is reported in
Table 4 and
Fig. 2. For 30-day mortality, both PACS+ model 1 (AUC, 0.828; 95% CI, 0.820–0.836) and PACS+ model 2 (AUC, 0.812; 95% CI, 0.805–0.818) demonstrated superior performance to the reference model, PACS (AUC, 0.722; 95% CI, 0.714–0.729). Similarly, for 7-day mortality, PACS+ model 1 (AUC, 0.859; 95% CI, 0.848–0.868) and PACS+ model 2 (AUC, 0.840; 95% CI, 0.832–0.848) outperformed PACS alone (AUC, 0.807; 95% CI, 0.795–0.819). However, for 2-day mortality, the predictive performance of PACS+ model 1 (AUC, 0.870; 95% CI, 0.856–0.882) and PACS+ model 2 (AUC, 0.855; 95% CI, 0.846–0.863) did not differ significantly from that of PACS (AUC, 0.858; 95% CI, 0.843–0.872).
Although PACS+ model 1 generally yielded lower sensitivities for 2-, 7-, and 30-day mortality than PACS and PACS+ model 2, its specificity was substantially higher. For example, for 30-day mortality, PACS+ model 1 yielded a sensitivity of 0.835 (95% CI, 0.820–0.849), compared with 0.990 (95% CI, 0.985–0.993) for PACS+ model 2 and 0.979 (95% CI, 0.972–0.984) for PACS, when triage levels P1 and P2 were classified as high acuity to predict mortality and levels P3 and P4 as lower acuity to predict nonmortality. However, specificity for PACS+ model 1 at 7-day mortality was substantially higher at 0.741 (95% CI, 0.739–0.744), whereas PACS and PACS+ model 2 both yielded specificities <0.250 (
Table 4).
Use of PACS+ model 1 resulted in a substantially greater proportion of patients being triaged to lower P3 and P4 (73.5%) than with PACS (24.3%) or PACS+ model 2 (22.9%). However, the percentage of patients with 30-day mortality assigned to lower P3 and P4 increased only slightly to 0.60% (426 of 71,464) with PACS+ model 1, compared with 0.23% (55 of 23,656) with PACS and 0.12% (27 of 22,242) with PACS+ model 2 (
Fig. 3,
Table 5). With PACS+ model 1, the percentages of patients with 2- and 7-day mortality among those triaged to P3 and P4 also increased only marginally. Among the different models, the proportion of patients triaged to P1 was substantially higher with PACS+ model 2 than with PACS or PACS+ model 1 (PACS+ model 2, 25.03%; PACS, 15.40%; PACS+ model 1, 14.50%). However, the percentage of patients triaged to P1 who experienced 30-day mortality was highest with PACS+ model 1 at 13.09% (1,845 of 14,093), followed by PACS+ model 2 at 8.75% (2,130 of 24,334) and PACS at 8.23% (1,232 of 14,968). Likewise, the percentage of patients triaged to P1 who experienced 7-day mortality was highest with PACS+ model 1 at 4.95% (697 of 14,093), followed by PACS at 4.34% (650 of 14,968), and then PACS+ model 2 at 3.48% (847 of 24,334).
Both PACS+ model 1 and PACS+ model 2 showed decreasing observed outcome proportions with decreasing triage priority. For example, as triage priority decreased from P1–P2 to P3–P4, the observed proportion of 30-day mortality also decreased (PACS+ model 1, 13.09% to 2.69% to 0.60%; PACS+ model 2, 8.75% to 0.84% to 0.12%).
Net benefit and decision curve analysis
As shown in
Fig. 4, for 2-day mortality, at threshold probabilities <0.20% and a prevalence of 0.44%, net benefit was maximized by PACS+ model 2. For 7-day mortality, at threshold probabilities <0.30% and a prevalence of 0.96%, net benefit was maximized by PACS+ model 2. For 30-day mortality, at threshold probabilities of 0% to 0.89% and a prevalence of 2.66%, net benefit was maximized by PACS+ model 2. Otherwise, PACS+ model 1 showed higher net benefit in all three scenarios.
DISCUSSION
Despite the potential of ML to enhance the quality of care in emergency medicine, clinical implementation remains limited [
27]. This study is among the first to incorporate an ML-based mortality prediction score into a triage system to improve the prediction of 30-day mortality among patients presenting to the ED. It is unique in that it does not attempt to replace or compete with the conventional triage system, but rather to serve as an adjunct to improve clinical decision-making. This approach allows us to leverage the strengths of both approaches, with ML providing data-driven insights and clinicians contributing contextual understanding and clinical acumen. In this study, we demonstrated that PACS+ model 1 and PACS+ model 2 showed superior predictive performance for 30- and 7-day mortality compared with PACS alone. These findings suggest that integrating 30-day SERP may be a feasible approach for identifying patients at high risk of 30-day mortality and may serve as a valuable aid to support clinical decision-making in the ED.
We note that PACS+ model 1 and PACS+ model 2 did not show a statistically significant difference from PACS in predicting 2-day mortality. This may be attributable to the markedly low 2-day mortality rate of 0.44%, which resulted in substantial class imbalance, with the positive class representing 2-day mortality being considerably underrepresented. One possible explanation for the enhanced performance of PACS+ model 1 and model 2 in predicting 7- and 30-day mortality is the incorporation of age and cancer history into 30-day SERP. Previous studies have shown that, among patients with similar presenting complaints and triage levels, older adults have higher mortality rates than younger adults [
28].
According to a 2015 report published by the State of Emergency Medical Care in Singapore Workgroup, approximately 7% of all ED patients in Singapore are triaged as P1, 40% as P2, 52% as P3, and 1% as P4 [
29]. However, in our study of ED attendees presenting in 2019, 15.4% were triaged to P1, 60.27% were triaged to P2, and 24.33% were triaged to P3 and P4 combined. This reflects a substantial increase in the proportion of patients triaged to high-priority levels. Concurrently, ED overcrowding has been widely reported, leading to prolonged waiting times and high hospital bed occupancy [
30]. This may hinder healthcare providers’ ability to deliver safe and timely emergency care, potentially contributing to adverse patient outcomes [
31]. Therefore, there is a need to rapidly identify patients who can be down-triaged and potentially referred to and managed in primary or urgent care settings, thereby alleviating the ED caseload. Our study showed that use of PACS+ model 1 resulted in a substantial increase in the proportion of patients triaged to lower P3 and P4 compared with PACS alone. Notably, among patients triaged to P1, the proportion with 30-day mortality increased substantially with PACS+ model 1 (13.09% vs. 8.23% with PACS), indicating improved identification of high-risk patients. Conversely, among those assigned to lower P3 and P4, 30-day mortality increased only modestly (0.60% with PACS+ model 1 vs. 0.23% with PACS). These findings suggest that PACS+ model 1 may enable safer down-triage of low-acuity patients, facilitating more efficient resource allocation toward critically ill patients. This reallocation may promote earlier interventions, enhance quality of care, and ultimately reduce mortality [
27,
32].
Conversely, PACS+ model 2 substantially increased the number of patients triaged to P1 compared with PACS+ model 1 and PACS, whereas the proportion of patients with 30-day mortality increased only marginally. Although PACS+ model 1 generally exhibited slightly lower sensitivities for 2-, 7-, and 30-day mortality than PACS and PACS+ model 2, it achieved markedly higher specificity. Previous studies have shown that over-triage is strongly associated with increased healthcare expenditure and inefficient allocation of scarce resources to patients with nonurgent conditions [
33]. This can exacerbate ED overcrowding, increase healthcare costs, and contribute to suboptimal outcomes for patients with truly emergent conditions [
34].
Because 30-day SERP uses only six variables that are routinely available during triage, it can be readily computed by trained healthcare workers, making it relatively quick and simple to integrate into the fast-paced ED workflow. Moreover, its interpretable nature may help clinicians understand the basis of its predictions, which is important for trust and clinical acceptance.
Limitations
There are several limitations to this study that should be acknowledged. First, because this was a single-center retrospective study, our analysis was a simulation-based reclassification limited to the available data and may not directly reflect the real-world impact of implementing PACS+ in practice. For example, our analysis relied on the initial triage assessment and did not account for dynamic changes in patient condition during the ED stay, which may result in subsequent up-triage or down-triage in routine clinical care. Moreover, triage performance is influenced by operational factors, such as staffing, monitoring capacity, and space, that cannot be evaluated using mortality outcomes alone. These workflow and safety implications were beyond the scope of this study and should be examined in future prospective or simulation-based research. Second, we excluded ED visits from noncitizens because their deaths are not registered in the Singapore death registry. This could introduce selection bias and limit generalizability to nonresident populations, which may differ in baseline health status, access to care, and follow-up patterns. We also excluded citizens with missing information, mainly triage priority level, which could potentially introduce selection bias. However, the excluded subset of citizens constituted <5% of the overall study cohort and demonstrated rates of 2-, 7-, and 30-day mortality comparable to those of the development and test cohorts. Furthermore, for the remaining variables, such as vital sign values, missing data were imputed using median values. Although this approach allowed us to retain all cases, it may have introduced bias. Third, 30-day mortality was chosen as the primary outcome to assess patient acuity. However, because ED outcomes often manifest acutely, this measure may not fully capture acute deterioration. Although other measures, such as patient diagnoses and interventions received, could be valuable for assessing triage acuity [
35], they were not available in the dataset at the time of analysis. Fourth, patient characteristics and clinical workflows were highly similar because both the training and test cohorts were derived from the same institution and data source. This homogeneity likely contributed to the comparable performance across datasets, but it may also limit generalizability. We believe that the model can be safely applied within the same hospital setting; however, external validation is required before it can be applied to other settings with different patient characteristics. Finally, triage decisions occur within a complex operational environment, and implementation of a new triage category may influence staffing demands, monitoring capacity, and patient flow. Furthermore, although PACS+ incorporates the interpretable SERP model, interpretability at the point of care may still be challenging. The model provides risk estimates, but triage nurses must translate these into actionable decisions and communicate them to clinicians or patients. Our study did not evaluate these workflow or safety implications; therefore, prospective or simulation-based studies are needed to assess the real-world impact of this new approach.
In this study, we developed two ways in which 30-day SERP, a validated clinical decision support tool, can be integrated into a conventional triage system. Our study demonstrated that PACS+ model 1 and model 2 showed superior performance in predicting both 7- and 30-day mortality compared with conventional PACS. In particular, use of PACS+ model 1 would substantially reduce the number of patients over-triaged to higher P1 and P2 without substantially increasing 2-, 7-, and 30-day mortality among patients in the lower P3 and P4 categories. These findings suggest that SERP integration improves 30-day mortality prediction and may offer meaningful clinical utility; however, prospective multicenter validation is required to confirm generalizability and real-world impact before large-scale clinical adoption.
NOTES
-
Author contributions
Conceptualization: MEHO, YWQF, KBKT; Data curation: YWQF; Formal analysis: YWQF, SFC; Funding acquisition: YO, MEHO, KBKT; Validation: YO, SFC; Visualization: YWQF; Writing–original draft: YWQF; Writing–review & editing: all authors. All authors read and approved the final manuscript.
-
Conflicts of interest
Marcus Eng Hock Ong is a scientific advisor to TIIM Healthcare SG and Global Healthcare SG. He is also a senior editor of this journal, but was not involved in the peer reviewer selection, evaluation, or decision process of this article. The authors have no other conflicts of interest to declare.
-
Funding
Yohei Okada has received a research grant from the Zoll Foundation and overseas scholarships from the Japan Society for Promotion of Science, the Fukuda Foundation for Medical Technology, the International Medical Research Foundation, and the Khoo Postdoctoral Fellowship Award (KPFA). Marcus Eng Hock Ong received grants from the Laerdal Foundation, Laerdal Medical, and Ramsey Social Justice Foundation for funding of the Pan-Asian Resuscitation Outcomes Study; funding from Laerdal Medical on an observation program to their Community CPR Training Centre Research Program in Norway; and has an advisory relationship with Global Healthcare SG, a commercial entity that manufactures cooling devices. These funds had no role in conducting this study.
-
Acknowledgments
The authors would like to express their gratitude to Associate Professor Liu Nan, Associate Professor at the Centre for Quantitative Medicine (CQM) and Programme in Health Services and Systems Research (HSSR), Duke-NUS Medical School, for his invaluable guidance and support throughout this research endeavor. The authors would also like to extend their heartfelt appreciation to all the members of the Health Services and Systems Research (HSSR) program and the Digital Medicine Lab for their continuous support throughout this project.
-
Data availability
The data analyzed in this study are available from the corresponding author upon reasonable request.
-
Additional information
This study was presented as a poster at the World Academic Conference of Emergency Medicine (WACEM) 2024, on November 6–8, 2024, in Rome, Italy.
Supplementary materials
Suppl. 1.
Definition of the four priority levels used in Patient Acuity Category Scale (PACS).
ceem-25-253-Suppl-1.pdf
Fig. 1.Study flowchart. Flow of patients presenting to the emergency department (ED) from January 1, 2018, to December 31, 2019. SGH, Singapore General Hospital; PACS, Patient Acuity Category Scale; SERP, Score for Emergency Risk Prediction; AUC, area under the receiver operating characteristic curve; ROC, receiver operating characteristic; NPV, negative predictive value; PPV, positive predictive value.
Fig. 2.Area under the receiver operating characteristic curves (AUCs) for (A) 2-, (B) 7- and (C) 30-day mortality using Patient Acuity Category Scale (PACS), PACS+ model 1, and PACS+ model 2 in the test cohort.
Fig. 3.Comparison of distribution and percentage of patients with 2-, 7- and 30-day mortality in each priority level between Patient Acuity Category Scale (PACS), PACS+ model 1, and PACS+ model 2. (A) Distribution of patients in each triage category and model. (B) 2-Day mortality. (C) 7-Day mortality. (D) 30-Day mortality.
Fig. 4.Decision curve analysis for outcomes of (A) 2-, (B) 7- and (C) 30-day mortality using the Patient Acuity Category Scale (PACS), PACS+ model 1, and PACS+ model 2 in the test cohort.
Table 1.
Table 1.
|
Initial PACS triage level |
PACS+ model 1 (30-day SERP score) |
|
≥26 (High risk) |
<26 (Low risk) |
|
P1 |
P1 |
P2 |
|
P2 |
P1 |
P3 |
|
P3 |
P2 |
P3 |
|
P4 |
P4 |
P4 |
Table 2.
Table 2.
|
Initial PACS triage level |
PACS+ model 2 (30-day SERP score) |
|
≥26 (High risk) |
<26 (Low risk) |
|
P1 |
P1 |
P1 |
|
P2 |
P1 |
P2 |
|
P3 |
P2 |
P3 |
|
P4 |
P4 |
P4 |
Table 3.Baseline characteristics of the development and test cohorts (n=194,400)
Table 3.
|
Characteristic |
Development cohort (n=97,188) |
Test cohort (n=97,212) |
|
Age (yr) |
58.97±18.41 |
59.42±18.40 |
|
Sex |
|
|
|
Male |
49,195 (50.6) |
48,955 (50.4) |
|
Female |
47,993 (49.4) |
48,257 (49.6) |
|
Race |
|
|
|
Chinese |
68,552 (70.5) |
69,423 (71.4) |
|
Malay |
12,679 (13.1) |
12,115 (12.5) |
|
Indian |
11,759 (12.1) |
11,308 (11.6) |
|
Others |
4,198 (4.3) |
4,366 (4.5) |
|
PACS triage category |
|
|
|
P1 |
19,302 (19.9) |
14,968 (15.4) |
|
P2 |
55,833 (57.4) |
58,588 (60.3) |
|
P3 |
21,846 (22.5) |
23,398 (24.1) |
|
P4 |
207 (0.2) |
258 (0.3) |
|
Shift time |
|
|
|
8:00–16:00 |
49,749 (51.2) |
49,602 (51.1) |
|
16:00–24:00 |
35,259 (36.3) |
35,527 (36.5) |
|
24:00–8:00 |
12,180 (12.5) |
12,083 (12.4) |
|
Vital sign |
|
|
|
Pulse (/min) |
83.73±17.27 |
82.39±17.17 |
|
Respiration (/min) |
18.00±1.89 |
17.17±1.74 |
|
SpO2 (%) |
97.76±4.19 |
97.75±4.54 |
|
Diastolic blood pressure (mmHg) |
75.33±13.57 |
73.35±14.00 |
|
Systolic blood pressure (mmHg) |
137.12±26.10 |
134.93±25.30 |
|
2-Day mortality |
497 (0.5) |
424 (0.4) |
|
7-Day mortality |
1,020 (1.1) |
936 (1.0) |
|
30-Day mortality |
2,707 (2.8) |
2,584 (2.7) |
|
ICU admission |
902 (0.9) |
817 (0.8) |
|
30-Day SERP score |
|
|
|
≥26 (High risk) |
15,680 (16.1) |
15,516 (16.0) |
|
<26 (Low risk) |
81,508 (83.9) |
81,696 (84.0) |
Table 4.Prediction performance of PACS, PACS+ model 1, and PACS+ model 2 in the test cohort
Table 4.
|
Outcome |
AUC (95% CI) |
Sensitivity (95% CI) |
Specificity (95% CI) |
NPV (95% CI) |
PPV (95% CI) |
|
2-Day mortality |
|
|
|
|
|
|
PACS |
0.858 (0.843–0.872) |
1.000 (0.991–1.000) |
0.244 (0.242–0.247) |
1.000 (1.000–1.000) |
0.006 (0.005–0.006) |
|
PACS+ model 1 |
0.870 (0.856–0.882) |
0.943 (0.917–0.963) |
0.738 (0.735–0.741) |
1.000 (1.000–1.000) |
0.016 (0.014–0.017) |
|
PACS+ model 2 |
0.855 (0.846–0.863) |
1.000 (0.991–1.000) |
0.230 (0.227–0.232) |
1.000 (1.000–1.000) |
0.006 (0.005–0.006) |
|
7-Day mortality |
|
|
|
|
|
|
PACS |
0.807 (0.795–0.819) |
0.991 (0.983–0.996) |
0.246 (0.243–0.248) |
1.000 (0.999–1.000) |
0.013 (0.012–0.013) |
|
PACS+ model 1 |
0.859 (0.848–0.868) |
0.910 (0.890–0.928) |
0.741 (0.739–0.744) |
0.999 (0.999–0.999) |
0.033 (0.031–0.035) |
|
PACS+ model 2 |
0.840 (0.832–0.848) |
0.997 (0.991–0.999) |
0.231 (0.228–0.234) |
1.000 (1.000–1.000) |
0.012 (0.012–0.013) |
|
30-Day mortality |
|
|
|
|
|
|
PACS |
0.722 (0.714–0.729) |
0.979 (0.972–0.984) |
0.249 (0.247–0.252) |
0.998 (0.997–0.998) |
0.034 (0.033–0.036) |
|
PACS+ model 1 |
0.828 (0.820–0.836) |
0.835 (0.820–0.849) |
0.751 (0.748–0.753) |
0.994 (0.993–0.995) |
0.084 (0.080–0.087) |
|
PACS+ model 2 |
0.812 (0.805–0.818) |
0.990 (0.985–0.993) |
0.235 (0.232–0.237) |
0.999 (0.998–0.999) |
0.034 (0.033–0.035) |
Table 5.Comparison of distribution and percentage of patients with 2-, 7-, and 30-day mortality in each priority level between PACS, PACS+ model 1, and PACS+ model 2 in the test cohort (n=97,212)
Table 5.
|
Outcome |
Triage level |
|
P1 |
P2 |
P3–P4 |
|
PACS (n=14,968, 15.4%) |
PACS+ model 1 (n=14,093, 14.5%) |
PACS+ model 2 (n=24,334, 25.0%) |
PACS (n=58,588, 60.3%) |
PACS+ model 1 (n=11,655, 12.0%) |
PACS+ model 2 (n=50,636, 52.1%) |
PACS (n=23,656, 24.3%) |
PACS+ model 1 (n=71,464, 73.5%) |
PACS+ model 2 (n=22,242, 22.9%) |
|
Admission |
11,677 (78.01) |
11,874 (84.26) |
19,253 (79.12) |
31,658 (54.04) |
8,035 (68.94) |
24,765 (48.91) |
4,910 (20.76) |
28,361 (39.69) |
4,254 (19.13) |
|
2-Day mortality |
349 (2.33) |
312 (2.21) |
400 (1.64) |
75 (0.13) |
88 (0.76) |
24 (0.05) |
0 (0) |
24 (0.03) |
0 (0) |
|
7-Day mortality |
650 (4.34) |
697 (4.95) |
847 (3.48) |
278 (0.47) |
155 (1.33) |
86 (0.17) |
8 (0.03) |
84 (0.12) |
3 (0.01) |
|
30-Day mortality |
1,232 (8.23) |
1,845 (13.09) |
2,130 (8.75) |
1,297 (2.21) |
313 (2.69) |
427 (0.84) |
55 (0.23) |
426 (0.60) |
27 (0.12) |
REFERENCES
- 1. Di Somma S, Paladino L, Vaughan L, Lalle I, Magrini L, Magnanti M. Overcrowding in emergency department: an international issue. Intern Emerg Med 2015;10:171-5.
- 2. Christ M, Grossmann F, Winter D, Bingisser R, Platz E. Modern triage in the emergency department. Dtsch Arztebl Int 2010;107:892-8.
- 3. Zachariasse JM, van der Hagen V, Seiger N, Mackway-Jones K, van Veen M, Moll HA. Performance of triage systems in emergency care: a systematic review and meta-analysis. BMJ Open 2019;9:e026471.
- 4. Mirhaghi A, Heydari A, Mazlom R, Ebrahimi M. The reliability of the Canadian Triage and Acuity Scale: meta-analysis. N Am J Med Sci 2015;7:299-305.
- 5. Fong RY, Glen WS, Mohamed Jamil AK, Tam WW, Kowitlawakul Y. Comparison of the Emergency Severity Index versus the Patient Acuity Category Scale in an emergency setting. Int Emerg Nurs 2018;41:13-8.
- 6. Fernandes M, Vieira SM, Leite F, Palos C, Finkelstein S, Sousa JM. Clinical decision support systems for triage in the emergency department using intelligent systems: a review. Artif Intell Med 2020;102:101762.
- 7. Jala S, Fry M, Elliott R. Cognitive bias during clinical decision-making and its influence on patient outcomes in the emergency department: a scoping review. J Clin Nurs 2023;32:7076-85.
- 8. Xie F, Ong ME, Liew JN, et al. Development and assessment of an interpretable machine learning triage tool for estimating mortality after emergency admissions. JAMA Netw Open 2021;4:e2118467.
- 9. Hong WS, Haimovich AD, Taylor RA. Predicting hospital admission at emergency department triage using machine learning. PLoS One 2018;13:e0201016.
- 10. Xie F, Zhou J, Lee JW, et al. Benchmarking emergency department prediction models with machine learning and public electronic health records. Sci Data 2022;9:658.
- 11. Fernandes M, Mendes R, Vieira SM, et al. Predicting Intensive Care Unit admission among patients presenting to the emergency department using machine learning and natural language processing. PLoS One 2020;15:e0229331.
- 12. Raita Y, Goto T, Faridi MK, Brown DF, Camargo CA, Hasegawa K. Emergency department triage prediction of clinical outcomes using machine learning models. Crit Care 2019;23:64.
- 13. Levin S, Toerper M, Hamrock E, et al. Machine-learning-based electronic triage more accurately differentiates patients with respect to clinical outcomes compared with the Emergency Severity Index. Ann Emerg Med 2018;71:565-74.
- 14. Yun H, Choi J, Park JH. Prediction of critical care outcome for adult patients presenting to emergency department using initial triage information: an XGBoost algorithm analysis. JMIR Med Inform 2021;9:e30770.
- 15. Tonekaboni S, Joshi S, McCradden MD, Goldenberg A. What clinicians want: contextualizing explainable machine learning for clinical end use. In: Doshi-Velez F, Fackler J, Jung K, et al., editors. Proceedings of the 4th Machine Learning for Healthcare Conference; 2019 Aug 9-10; Ann Arbor, MI, USA. PMLR 2019;106:359-80.
- 16. Rudin C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat Mach Intell 2019;1:206-15.
- 17. Yu JY, Xie F, Nan L, et al. An external validation study of the Score for Emergency Risk Prediction (SERP), an interpretable machine learning-based triage score for the emergency department. Sci Rep 2022;12:17466.
- 18. Liu N, Xie F, Siddiqui FJ, et al. Leveraging large-scale electronic health records and interpretable machine learning for clinical decision making at the emergency department: protocol for system development and validation. JMIR Res Protoc 2022;11:e34201.
- 19. Blomaard LC, Speksnijder C, Lucke JA, et al. Geriatric screening, triage urgency, and 30-day mortality in older emergency department patients. J Am Geriatr Soc 2020;68:1755-62.
- 20. Tschoellitsch T, Seidl P, Böck C, et al. Using emergency department triage for machine learning-based admission and mortality prediction. Eur J Emerg Med 2023;30:408-16.
- 21. Zelis N, Buijs J, de Leeuw PW, van Kuijk SM, Stassen PM. A new simplified model for predicting 30-day mortality in older medical emergency department patients: the rise up score. Eur J Intern Med 2020;77:36-43.
- 22. Jones S, Moulton C, Swift S, et al. Association between delays to patient admission from the emergency department and all-cause 30-day mortality. Emerg Med J 2022;39:168-73.
- 23. Kaeppeli T, Rueegg M, Dreher-Hummel T, et al. Validation of the Clinical Frailty Scale for prediction of thirty-day mortality in the emergency department. Ann Emerg Med 2020;76:291-300.
- 24. Shen Y, Lee LH. Improving the wait time to consultation at the emergency department. BMJ Open Qual 2018;7:e000131.
- 25. Xie F, Chakraborty B, Ong ME, Goldstein BA, Liu N. AutoScore: a machine learning-based automatic clinical score generator and its application to mortality prediction using electronic health records. JMIR Med Inform 2020;8:e21798.
- 26. Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making 2006;26:565-74.
- 27. Yin J, Ngiam KY, Teo HH. Role of artificial intelligence applications in real-life clinical practice: systematic review. J Med Internet Res 2021;23:e25759.
- 28. Ginsburg AD, Oliveira J E Silva L, Mullan A, et al. Should age be incorporated into the adult triage algorithm in the emergency department? Am J Emerg Med 2021;46:508-14.
- 29. State of Emergency Medical Care in Singapore Workgroup. State of emergency medical care in Singapore: addressing challenges and charting directions. College of Emergency Physicians, Academy of Medicine, Singapore; 2015. Co-published by the Society for Emergency Medicine in Singapore.
- 30. Schoenenberger LK, Bayer S, Ansah JP, et al. Emergency department crowding in Singapore: insights from a systems thinking approach. SAGE Open Med 2016;4:2050312116671953.
- 31. Sprivulis PC, Da Silva JA, Jacobs IG, Frazer AR, Jelinek GA. The association between hospital overcrowding and mortality among patients admitted via Western Australian emergency departments. Med J Aust 2006;184:208-12.
- 32. Hoot NR, Aronsky D. Systematic review of emergency department crowding: causes, effects, and solutions. Ann Emerg Med 2008;52:126-36.
- 33. Hinson JS, Martinez DA, Schmitz PS, et al. Accuracy of emergency department triage using the Emergency Severity Index and independent predictors of under-triage and over-triage in Brazil: a retrospective cohort analysis. Int J Emerg Med 2018;11:3.
- 34. Calder LA, Forster AJ, Stiell IG, et al. Mapping out the emergency department disposition decision for high-acuity patients. Ann Emerg Med 2012;60:567-76.e4.
- 35. Johansson A, Ekwall A, Forberg JL, Ekelund U. Development of outcomes for evaluating emergency care triage: a Delphi approach. Scand J Trauma Resusc Emerg Med 2023;31:10.
, Yohei Okada1,2,3,*, Stephanie Fook-Chong2, Yilin Ning4, Kenneth Boon Kiat Tan1,5, Marcus Eng Hock Ong2,5
